PaperlayerView exact revision on arXiv

Source-linked AI summary

Lion: Adversarial Distillation of Proprietary Large Language Models

Yuxin Jiang, Chunkit Chan, Mingyang Chen, Wei Wang

arXiv:2305.12870v2cs.CL

TL;DR

Existing proprietary-to-open LLM distillation is largely unidirectional and lacks feedback about instructions where the student struggles. The paper introduces a three-stage adversarial loop that uses the teacher to identify and generate hard instructions, and reports that Lion-13B approaches ChatGPT on open-ended generation while outperforming Vicuna-13B on reasoning tasks. The method uses 70k examples but remains limited by reasoning, dialogue, computational, and cost constraints.

  • Problem

    Prior methods transfer knowledge from proprietary teachers to open students without feeding back the hard instructions where student performance falls short.

    Source: §1 · pp. 1–2

  • Method

    The framework prompts a proprietary LLM to support imitation, discriminate hard instructions, and generate new hard instructions in an iterative adversarial loop.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    Lion-13B approximates ChatGPT’s open-ended generation capabilities and largely outperforms Vicuna-13B on reasoning tasks after training on 70k data.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework provides a feedback-based approach for transferring proprietary LLM knowledge to compact open-source students and can be adapted to other proprietary LLMs.

    Source: §1 · p. 2, §7 · p. 9

    Lion still lags behind ChatGPT on intricate reasoning, struggles with multi-turn conversations because its training data lacks dialogues, and faces sequence-length constraints from limited computational resources.

    Source: Limitations and Discussions · p. 9

Abstract

from arXiv · show

The practice of transferring knowledge from a sophisticated, proprietary large language model (LLM) to a compact, open-source LLM has garnered considerable attention. Previous works have focused on a unidirectional knowledge distillation way by aligning the responses of the student model with those of the teacher model to a set of instructions. Nevertheless, they overlooked the possibility of incorporating any reciprocal "feedback"--identifying challenging instructions where the student model's performance falls short--to boost the student model's proficiency iteratively. To this end, we propose a novel adversarial distillation framework for a more efficient knowledge transfer. Leveraging the versatile role adaptability of LLMs, we prompt the teacher model to identify "hard" instructions and generate new "hard" instructions for the student model, creating a three-stage adversarial loop of imitation, discrimination, and generation. By applying this adversarial framework, we successfully transfer knowledge from ChatGPT to a student model (named Lion), using a mere 70k training data. Our results show that Lion-13B not only achieves comparable open-ended generation capabilities to ChatGPT but surpasses conventional state-of-the-art (SOTA) instruction-tuned models like Vicuna-13B by 55.4% in challenging zero-shot reasoning benchmarks such as BIG-Bench Hard (BBH) and 16.7% on AGIEval. Code and model can be found at https://github.com/YJiangcm/Lion.

1 Introduction

The paper proposes adversarial knowledge distillation that feeds student weaknesses back to a proprietary teacher, enabling iterative training on increasingly hard instructions. Applied to ChatGPT and LLaMA, the framework trains Lion on 70k data and reports strong performance against open-source instruction-tuned baselines.

Motivation

  • Previous distillation methods transfer knowledge one way from teacher to student without feedback about difficult instructions.The paper defines feedback as identifying examples where the student’s performance falls short.Source: §1 · pp. 1–2

Contribution

  • The method applies adversarial knowledge distillation to LLMs by using a proprietary model’s role adaptability instead of requiring teacher weights or gradients.The paper describes this as its first attempt to adopt adversarial knowledge distillation for large language models.Source: §1 · p. 2

Method

  • The framework uses a three-stage loop: imitation aligns responses, discrimination identifies hard instructions, and generation creates new hard instructions.The proprietary teacher is prompted to act as both referee and generator because its weights and gradients are inaccessible.Source: §1 · p. 2

Experimental setup

  • The framework transfers ChatGPT knowledge to LLaMA through three AKD iterations initialized with Alpaca instructions, producing 70K training examples for Lion.ChatGPT is accessed through the OpenAI API using the gpt-3.5-turbo model.Source: §1 · p. 2

Results

  • Lion-13B approximates ChatGPT on open-ended generation and largely outperforms Vicuna-13B on reasoning tasks using 70k unannotated training examples.The evaluation covers open-ended generation and reasoning datasets totaling 40 subtasks.Source: §1 · p. 2

Contribution

  • The framework is designed to adapt beyond ChatGPT to other proprietary large language models.The paper presents this adaptability as a consequence of the framework’s versatile role design.Source: §1 · p. 2

2 Related Work

Related work covers instruction-following language models and knowledge distillation. It highlights the difficulty of reproducing proprietary instruction data and adapting existing distillation methods to black-box, sequence-generating LLMs.

2.1 Instruction-Following Language Models

  • Instruction tuning enables language models to generalize to unseen tasks from natural-language instructions, but simple task-specific instructions do not capture diverse real-world user intent.Human-crafted instruction datasets used by InstructGPT, ChatGPT, and GPT-4 are not open-sourced.Source: §2.1 · pp. 2–3
  • Alpaca, Vicuna, and WizardLM generate high-quality instructions to fine-tune LLaMA and approach ChatGPT’s performance despite proprietary instruction data.These efforts address the lack of access to the diverse human-annotated instructions used by proprietary models.Source: §2.1 · pp. 2–3

2.2 Knowledge Distillation

  • Knowledge distillation compresses a large teacher into a compact student, but data-free methods commonly require teacher weights or gradients or rely on proxy data and generators.Existing approaches include synthetic data generation, teacher inversion, and zero-order gradient estimation.Source: §2.2 · p. 3
  • Adapting these distillation methods to LLMs is difficult because text responses are sequences rather than continuous softmax vectors and instruction generators must reproduce diverse data distributions.The passage identifies these as two main hurdles for applying prior methods to the paper’s setting.Source: §2.2 · p. 3

3 Methodology

The methodology iteratively distills a proprietary teacher into a compact student by aligning responses, identifying performance gaps, and generating new hard instructions. Its adversarial loop uses multiple teacher roles and expanding data pools to target difficult examples over successive iterations.

3 Methodology

  • Hard samples target instructions where teacher–student performance gaps are relatively large, because these samples dominate expected model discrepancy.As mastered samples become easy, the framework continuously generates new hard samples to sustain training pressure.Source: §3 · p. 3
  • The framework iterates through imitation, discrimination, and generation to align the student with the teacher, identify hard instructions, and create new challenging samples.The proprietary LLM serves as teacher, referee, and generator, while the student is initialized from a foundation LLM.Source: §3 · p. 3, §3.1 · p. 3, Figure 2 · p. 4

3.2 Imitation Stage

  • The imitation stage fine-tunes the student on teacher-generated responses using an autoregressive language-modeling objective.Instructions from the Train Pool are forward propagated through the teacher to construct instruction–response training data.Source: §3.2 · p. 4

3.3 Discrimination Stage

  • The discrimination stage evaluates teacher and student responses in an expanding Cache Pool and labels instructions hard when their score difference satisfies d_i ≥ τ.The referee averages scores from swapped response positions to reduce positional bias; experiments use τ = 1.0.Source: §3.3 · p. 4
  • Identified hard instructions concentrate more on complex tasks such as math and coding than the broader Cache Pool distribution.Figure 3 compares instruction distributions using common root verbs and direct noun objects.Source: §3.3 · p. 5, Figure 3 · p. 5

3.4 Generation Stage

  • The generator creates new instructions matching hard examples’ domains and task types, while also generating long-tailed easy examples to preserve diversity and mitigate catastrophic forgetting.New instructions require ROUGE-L overlap below 0.7 with existing Cache Pool instructions, and generated hard and easy instructions are maintained at a 1:1 ratio.Source: §3.4 · p. 5

3.5 Min-Max Game Interpretation

  • The min-max interpretation alternates minimizing student–teacher discrepancy on hard samples with generating samples that maximize this discrepancy as the student improves.Ideally, training reaches equilibrium when the referee can no longer distinguish student and teacher responses.Source: §3.5 · p. 5

4 Experiments Setting

The experiments evaluate Lion across open-ended generation and zero-shot reasoning, using established instruction-following and reasoning benchmarks. They compare against several open-source and proprietary baselines under specified evaluation and training settings.

4.1 Datasets

  • The evaluation covers writing, coding, commonsense, mathematics, and logical reasoning through open-ended generation and reasoning datasets.The datasets are divided into two broad categories: open-ended generation and reasoning.Source: §4.1 · p. 5

Vicuna-Instructions

  • Vicuna-Instructions contains 80 questions across 9 task categories and is evaluated by GPT-4 relative scoring against ChatGPT.Candidate performance is computed as the percentage of the reference model’s total score.Source: Vicuna-Instructions · p. 6

4.1.2 Reasoning Datasets

  • AGIEval contributes 2,546 English multiple-choice questions across 8 tasks, while BBH contributes 5,511 samples across 23 multiple-choice tasks.Both benchmarks assess reasoning in a zero-shot setting without exemplars or Chain-of-Thought, using exact-match accuracy.Source: §4.1.2 · p. 6

4.2 Baselines

  • The baselines include LLaMA, Alpaca, WizardLM, Vicuna, and ChatGPT, with comprehensive comparison against Vicuna because of its strong open-source leaderboard standing.The baseline descriptions are provided in the appendix.Source: §4.2 · p. 6

4.3 Implementation Details

  • Lion is trained from pretrained LLaMA using three iterations and 6K newly generated instructions per iteration, yielding 70K training examples.The setup is designed for comparison with current SOTA baselines, including WizardLM and Vicuna.Source: §4.3 · p. 6
  • Inference uses temperature 0.7 and maximum generation length 1024 for Lion and ChatGPT, while other models follow their original papers’ settings.The proprietary model is configured through role-specific API hyperparameters described in the appendix.Source: §4.3 · p. 6

5 Experimental Results

Lion performs strongly against ChatGPT and Vicuna across open-ended generation and reasoning evaluations. Its advantages over Vicuna are especially pronounced on challenging reasoning benchmarks, although it remains below ChatGPT on average reasoning capability.

5.1 Results for Open-ended Generation

  • In Table 1, Lion-7B and Lion-13B remarkably outperform their counterparts under two evaluation settings in relative response quality against ChatGPT, using GPT-4 as referee/rater on Vicuna-Instructions.Source: §5.1 · p. 6, Table 1 · p. 7
  • Lion-13B slightly surpasses ChatGPT in generic, knowledge, common-sense, and counterfactual categories, while reaching at least 32.32% relative score in math.It also exceeds most baselines in coding generation.Source: §5.1 · p. 7, Figure 4 · p. 7

5.2 Results for Reasoning

  • Over 16% average relative improvement over Vicuna is achieved by Lion on AGIEval, while Lion-13B retains 72.5% of ChatGPT’s reasoning capability.The comparison covers zero-shot English multiple-choice questions.Source: §5.2 · p. 7
  • Around 50% average improvement over Vicuna is achieved on BIG-Bench Hard, with over 100% gains on several data, semantic, logical, geometric, and positional reasoning tasks.Lion reaches nearly 74% of ChatGPT’s average BBH ability and surpasses ChatGPT on several tasks.Source: §5.2 · p. 7

6 Analyses

The analyses examine instruction-hardness settings, training dynamics, and qualitative responses. They indicate that balanced hard-example generation and early adversarial iterations are especially useful, while case studies show stronger responses on selected tasks.

6.1 Ablation Studies

  • Threshold values between 1.0 and 1.5 are optimal across datasets; increasing the threshold from 1.0 to 2.0 gradually degrades performance as hard-instruction diversity decreases.The method is not highly sensitive to a large threshold value.Source: §6.1 · p. 7, Table 4 · p. 9
  • Across three datasets, increasing the hard-to-easy instruction ratio generally improves average performance; the study spans ratios from 1:0 (all hard) to 0:1 (all easy).Source: §6.1 · p. 7

6.2 The Learning Dynamics of Lion

  • Performance on AGIEval and BBH improves consistently across training iterations, with the largest capability increase occurring in the first iteration.The result supports the usefulness of identifying challenging example patterns.Source: §6.2 · p. 8, Figure 5 · p. 9

6.3 Case Studies

  • In qualitative examples, Lion and ChatGPT alone provide the correct answer and reasoning steps for a math instruction.For a counterfactual biology scenario, Lion supplies more detail and explores more possible implications than ChatGPT.Source: §6.3 · pp. 8–9

7 Conclusion

The paper introduces adversarial knowledge distillation for transferring proprietary-LLM capabilities to compact open-source models. Its three-stage feedback loop is intended to iteratively refine Lion and provide an open-source reference point for ChatGPT-level instruction following.

7 Conclusion

  • The framework addresses unidirectional distillation by incorporating teacher-generated feedback through imitation, discrimination, and generation stages.The teacher identifies hard instructions and generates new hard instructions for the student.Source: §7 · p. 9
  • Iterative hard-instruction generation is used to bootstrap the student model’s proficiency efficiently.The resulting model is named Lion.Source: §7 · p. 9
  • Lion is proposed as an open-source instruction-following baseline for reflecting ChatGPT’s performance.The stated role is especially relevant to the open-source community.Source: §7 · p. 9

Limitations and Discussions

Lion improves on prior imitation models but remains constrained by reasoning ability, training cost, evaluation reliability, and sequence length.

Limitations and Discussions

  • Lion still lags behind ChatGPT on intricate reasoning tasks, although it improves over previous imitation models.The authors frame adversarial knowledge distillation as a more effective transfer approach while acknowledging this remaining gap.Source: Limitations and Discussions · p. 9
  • Lion’s maximum sequence length is 1024, limiting its ability to handle long documents.Source: Limitations and Discussions · p. 9
  • 450k API requests and nearly $900 are required to train one student model, with iterative updates slowing iteration compared with one-time fine-tuning.The method uses roughly 70% of WizardLM’s API usage but incurs considerable expense.Source: Limitations and Discussions · pp. 9–10
  • GPT-4-based automated evaluation remains immature and potentially inaccurate because language models can generate hallucinated information.The authors also note that different tasks require different expertise, complicating comprehensive evaluation.Source: Limitations and Discussions · p. 10

Ethics Statement

The ethics discussion highlights inherited risks, licensing constraints, and the absence of human-feedback-based safety training in Lion.

Ethics Statement

  • Lion may inherit toxicity, biases, or privacy issues from its larger teacher model.Source: Ethics Statement · p. 10
  • Lion’s weights are intended for research use and must be used with LLaMA weights under Meta’s license agreement.Users are responsible for acquiring and using LLaMA according to that agreement.Source: Ethics Statement · p. 10
  • Unlike ChatGPT, Lion does not use human feedback to mitigate undesirable behaviors and may pose risks if used maliciously.The authors propose future RLHF integration and note Meta’s access application process as a distribution-control option.Source: Ethics Statement · p. 10

A Data Statistics

The paper reports dataset statistics and compares Lion with open-source and proprietary baselines spanning instruction-following and chatbot systems.

A Data Statistics

B Baselines •

  • LLaMA comprises foundation models ranging from 7B to 65B parameters and is reported to outperform GPT-3 across multiple benchmarks.Source: B Baselines • · p. 13
  • Alpaca is based on LLaMA and fine-tuned on 52K instruction-following examples generated by querying text-davinci-003.It is described as compact and cost-effective to reproduce.Source: B Baselines • · pp. 13–14
  • WizardLM expands Alpaca’s 52K examples into 250K more intricate instructions, selecting 70K examples to fine-tune LLaMA.Source: B Baselines • · p. 14
  • Vicuna is based on LLaMA and fine-tuned on 70K user-shared ShareGPT conversations, achieving competitive performance against proprietary models.Source: B Baselines • · p. 14
  • ChatGPT is presented as a proprietary chatbot built on powerful LLMs and fine-tuned with supervised and reinforcement learning methods using human feedback.Source: B Baselines • · p. 14

C Implementation Details

Implementation uses multi-GPU training, role-specific API settings, and prompt templates that operationalize the teacher, referee, and generator roles.

C Implementation Details

D Prompt Templates for Our Adversarial Distillation Framework

E Case Studies

The case studies compare model responses on open-ended generation and reasoning tasks, using prompt templates and tabulated examples. They include critical-thinking advice, arithmetic, date reasoning, and benchmark questions evaluated by correctness.

E Case Studies

  • The case studies compare Lion-13B, LLaMA-13B, Alpaca-13B, Vicuna-13B, and ChatGPT on open-ended generation examples.The examples are presented in Tables 14–16.Source: E Case Studies · p. 14, Table 14 · p. 17, Table 15 · p. 18
  • Prompt templates define how ChatGPT generates responses, compares two responses, and produces hard or easy instructions.These templates appear in Tables 10–13.Source: Table 10 · p. 15, Table 11 · p. 15, Table 12 · p. 15
  • The critical-thinking examples recommend analyzing information, asking questions, evaluating evidence, seeking diverse perspectives, and weighing decision outcomes.The examples also emphasize learning from mistakes and using critical thinking for informed decisions.Source: E Case Studies · p. 19

LLaMA

  • The LLaMA examples include broad critical-thinking advice and benchmark-style questions whose responses are marked correct or incorrect.The benchmark cases span AGIEval and BIG-Bench Hard tasks.Source: E Case Studies · p. 14, LLaMA · p. 19, Table 16 · p. 19
  • The cited cases include an arithmetic-reduction question, a navigation instruction sequence, a boolean-expressions case, and a data-understanding case, represented in Tables 17–20.Source: LLaMA · p. 20, Table 17 · p. 20, Table 18 · p. 20
  • The date-reasoning example identifies 12/25/1937 as the correct answer for the date tomorrow on Christmas Eve 1937.The explanation treats Christmas Day as one day after Christmas Eve and rejects the other listed dates.Source: LLaMA · p. 21
Loading 2305.12870v2…