Source-linked AI summary
The False Promise of Imitating Proprietary LLMs
Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, Dawn Song
TL;DR
The paper asks whether fine-tuning weaker open-source LMs on proprietary-model outputs can cheaply close the capabilities gap. It evaluates ChatGPT imitators across model sizes, data sources, imitation-data amounts, human and GPT-4 judgments, and NLP benchmarks, finding that imitation mainly improves style and instruction adherence while closing little of the broader gap. The authors therefore favor developing stronger base models, while noting that imitation may still reduce the need for annotated fine-tuning data when the base model is sufficiently strong.
Problem
The paper examines whether model imitation can cheaply transfer proprietary LMs’ capabilities to weaker open-source models and make them competitive with closed systems.
Method
The authors train ChatGPT imitators across base-model sizes, data sources, and imitation-data amounts, evaluating them with human and GPT-4 comparisons plus MMLU, Natural Questions, and HumanEval accuracy.
Results
Imitation improves style, persona, and instruction adherence but closes little to none of the gap on harder axes such as factuality, coding, and problem solving.
Takeaways & Limitations
Improving open-source LMs should prioritize stronger base models over fine-tuning on additional imitation data, though imitation can replace expensive fine-tuning annotation with a sufficiently strong base LM.
Takeaways & Limitations
Broadly matching ChatGPT with weaker base models may require enormous, highly diverse, high-quality imitation data near pretraining scale, whose feasibility and value remain unclear.
Abstract
from arXiv · showhide
An emerging method to cheaply improve a weaker language model is to finetune it on outputs from a stronger model, such as a proprietary system like ChatGPT (e.g., Alpaca, Self-Instruct, and others). This approach looks to cheaply imitate the proprietary model's capabilities using a weaker open-source model. In this work, we critically analyze this approach. We first finetune a series of LMs that imitate ChatGPT using varying base model sizes (1.5B--13B), data sources, and imitation data amounts (0.3M--150M tokens). We then evaluate the models using crowd raters and canonical NLP benchmarks. Initially, we were surprised by the output quality of our imitation models -- they appear far better at following instructions, and crowd workers rate their outputs as competitive with ChatGPT. However, when conducting more targeted automatic evaluations, we find that imitation models close little to none of the gap from the base LM to ChatGPT on tasks that are not heavily supported in the imitation data. We show that these performance discrepancies may slip past human raters because imitation models are adept at mimicking ChatGPT's style but not its factuality. Overall, we conclude that model imitation is a false promise: there exists a substantial capabilities gap between open and closed LMs that, with current methods, can only be bridged using an unwieldy amount of imitation data or by using more capable base LMs. In turn, we argue that the highest leverage action for improving open-source models is to tackle the difficult challenge of developing better base LMs, rather than taking the shortcut of imitating proprietary systems.
1 Introduction
The paper critically evaluates whether imitation can make weaker open-source LMs competitive with proprietary systems. Although imitation models appear competitive in human ratings, targeted evaluations show that they close little to none of the capability gap outside well-supported tasks.
- Powerful proprietary LMs raise whether leading systems will remain closed or become freely distributed and modifiable.
- The study trains ChatGPT imitators across model sizes, base models, data sources, and 0.3M–150M-token imitation datasets, then evaluates them with humans, GPT-4, and NLP benchmarks.
- Imitation models improve instruction following and appear similar to ChatGPT, with their best outputs rated competitively by human and GPT-4 evaluators.
- Imitation closes little to none of the LLaMA–ChatGPT gap on targeted evaluations; broad data offers no Natural Questions benefit, whereas task-specific data improves that task.
- Crowdworkers may favor imitation models because confident, well-structured style masks weak factuality, especially without domain expertise or substantial review time.
- Improving open-source models requires stronger base capabilities rather than simply adding imitation data, although imitation can reduce the need for annotated fine-tuning data with a sufficiently strong base LM.
2 What is Model Imitation?
Model imitation collects outputs from a proprietary API and uses them to fine-tune an open-source LM. The paper distinguishes task-specific imitation from broad coverage and evaluates whether this strategy can support claims of near parity.
- Model imitation uses outputs collected from a proprietary LM’s black-box API to fine-tune an open-source model toward comparable performance.
- Imitation models can support academic research, competing commercial services, or malicious applications.
- Task-specific imitation targets one domain or task, while broad-coverage imitation seeks to reproduce the target model across behaviors, domains, and tasks.
- Broad coverage requires extremely diverse data and generalization across many held-out examples, making it substantially more challenging than local imitation.
- The paper critically evaluates prior near-parity claims by varying data collection, data amounts, base LMs, and evaluation methods.
3 Building Imitation Datasets
The paper builds both task-specific and broad-coverage imitation datasets by querying ChatGPT with curated or synthetic inputs and by collecting publicly shared conversations. The broad-coverage mixture is diverse, including multilingual and coding examples.
- Imitation datasets begin with curated inputs or synthetic examples generated by prompting language models to create additional task-relevant queries.
- For Natural Questions, the authors seed ten validation QA pairs and use ChatGPT to generate 6,000 similar but distinct single-turn examples.
- Broad-coverage data is collected from publicly posted ChatGPT interactions, avoiding direct API interaction while providing large and diverse examples.
- ShareGPT-Mix combines approximately 50K filtered ShareGPT dialogues, roughly 27K HC3 responses, and Discord ChatGPT Bot data.
- ShareGPT-Mix has high instruction diversity, with average nearest-query BLEU similarity of 8% versus 61% for similarly sized Super-NaturalInstructions data, and includes multilingual and coding tasks.
- GPT-4 ratings remain relatively flat as imitation data increases but rise consistently with base-model scale.
4 Main Results
The study finds that broad ChatGPT imitation improves instruction-following and stylistic similarity, but generally fails to transfer broad factual and task capabilities. Targeted imitation can substantially improve a specific behavior, while scaling the base LM is more effective than adding broad imitation data.
- Crowdworker and GPT-4 Evaluations: Crowdworkers and GPT-4 rated the strongest imitation outputs as competitive with ChatGPT, especially as model size increased.The models were notably better than their base models at following instructions and producing on-task responses.
- Targeted Automatic Evaluations: Broad-coverage imitation models do not improve, and can regress, on most benchmarks as imitation data increases, including factual knowledge, coding, and problem solving.The reported gap between LLaMA and ChatGPT closes little to none on targeted automatic evaluations, particularly when tasks receive little support in training data.
- Targeted Automatic Evaluations: Targeted NQ-synthetic imitation substantially improves performance over LLaMA and quickly approaches ChatGPT accuracy on Natural Questions.The contrast with ShareGPT-Mix shows that distilling a specific behavior is more feasible than broadly matching ChatGPT’s capabilities.
- Style Versus Factuality: Imitation models match ChatGPT’s length, word choice, authoritative tone, and low-level structure, but retain weak factuality and can produce confident, completely incorrect answers.This stylistic similarity helps explain why human evaluations appear more favorable than benchmark results.
- Safety and Toxicity: Imitation also transfers ChatGPT’s safety behavior: outputs become less toxic on RealToxicityPrompts as more imitation data is added.The models learn to abstain in a manner similar to ChatGPT, making imitation useful when a powerful base LM needs behavioral alignment without expensive annotations.
5 Discussion
The discussion argues that finetuning on imitation data mainly improves style and knowledge extraction rather than underlying capabilities. It also identifies business implications, technical limitations, and possible extensions of the findings.
- Finetuning as a simple knowledge extractor: Modest finetuning provides little to no improvement in an LM’s knowledge or capabilities because base knowledge is largely unaffected.The authors characterize finetuning as a lightweight method for extracting a model’s existing knowledge.
- Should you be worried about imitation?: Imitation is limited when there is a large gap between the base and target LM, making capability gaps important for maintaining competitive advantages.Companies relying on off-the-shelf LMs with proprietary finetuning data may be comparatively more vulnerable to imitation.
- Potential confounders to our findings: ChatGPT’s unknown pre-training data creates a potential confounder because evaluated tasks may have been contaminated into its training data.Such contamination could inflate ChatGPT’s accuracy numbers.
- Potential confounders to our findings: The study uses supervised learning on target outputs and relatively simple data-collection methods, while RLHF, constitutional AI, or active learning might improve imitation.These possibilities are presented as unresolved alternatives rather than evaluated conclusions.
- Implications for other forms of model imitation: The same failure modes may affect vision-language models and other systems that incorporate ChatGPT or GPT-4 outputs during training or annotation.The authors present this as a likely implication for related approaches.
- Technical limitations of model imitation: Imitation models inherit proprietary systems’ weaknesses and biases, cannot directly improve closed-company design choices, and are roughly upper-bounded by the target model.Their proprietary black-box training pipelines also make some scientific questions difficult to answer.
6 Related Work
Related work places model imitation alongside distillation and prior model-stealing research. The paper distinguishes its setting by focusing on proprietary systems whose stronger pre-trained base models are hidden behind APIs.
- Model distillation: Model imitation resembles distillation but lacks access to the teacher’s training data, architecture, and hyperparameters.Distillation can also use probability-based training objectives, whereas imitation relies on API outputs without equivalent teacher knowledge.
- Past work on model imitation: Prior work demonstrates model imitation across domains including language classifiers and machine translation systems.Those studies commonly consider models trained from scratch, making internal training data the main proprietary component.
- Past work on model imitation: This paper’s setting additionally treats systems like ChatGPT as proprietary because their internal pre-trained LMs are stronger than available open-source models.The hidden strength of the base model is therefore distinct from proprietary fine-tuning data alone.
- Defending against model imitation: Existing defenses include detecting imitation or slowing model stealing by sacrificing some performance.The authors note that current methods often impose tradeoffs too severe for practical deployment.
7 Conclusion and Future Work
The conclusion finds that imitation reproduces style, persona, and instruction adherence more readily than factuality, coding, or problem solving. It therefore highlights both the strategic value of strong base models and unresolved questions about human evaluation and open-source improvement.
- 7 Conclusion and Future Work: Imitation improves style, persona, and instruction adherence but falls short on factuality, coding, and problem solving.The conclusion frames these as more challenging capability axes.
- 7 Conclusion and Future Work: Equally competent base LMs can readily mimic one another’s persona and behavior without expensive annotated finetuning data.This conclusion concerns behavioral imitation rather than matching all capabilities.
- 7 Conclusion and Future Work: Crowdworker evaluations have difficulty distinguishing imitation models from proprietary ones despite clear differences between them.The authors leave open how to probe powerful LLM utility cheaply and quickly.
- 7 Conclusion and Future Work: The gap between LLaMA and ChatGPT and imitation’s insufficiencies leave open questions about scaling models, improving pre-training data, and developing new pretraining methods.These are proposed directions rather than completed findings.
A Additional Details on Imitation Data
The appendix describes how synthetic Natural Questions imitation data are constructed from curated validation examples and prompting templates. It also identifies the relevant seed-example and template tables and points to broader dataset category breakdowns.
- Task-specific imitation: NQ-synthetic data begin with seed examples curated from the Natural Questions validation set.The seed examples are identified in Table 3.
- Task-specific imitation: New question-answer pairs are generated by prompting with five randomly sampled seed pairs, using temperature 1.0 and discarding duplicates.The prompting template is provided in Table 4.
- Additional dataset details: Figure 6 provides examples from ShareGPT-Mix, while Table 5 breaks down its categories.The passage identifies the resources but does not report a specific category comparison.
B Amazon Mechanical Turk Interface
The study compares imitation-model outputs with ChatGPT using crowd-based pairwise judgments. The evaluated ShareGPT inputs are varied across task categories, and the rating procedure averages three unique judgments per example.
- Human evaluation procedure: Evaluators compare an imitation model’s output with ChatGPT’s output for the same randomly selected instruction.They choose which response is better based on overall subjective quality, with system order randomized.
- Human evaluation procedure: Each evaluation example receives 3 unique ratings, with scores averaged across examples.
- Evaluation data: ShareGPT inputs span categories including open-ended generation, brainstorming, text extraction, coding, and multilingual queries.A manual review of 50 random ShareGPT queries found highly varied task instructions.
- Human evaluation procedure: The human-rating experiments cost roughly $5000 including service fees, with evaluators paid about $15 per hour.
C GPT-4 evaluations
The GPT-4 evaluation asks for preference rankings between ChatGPT and imitation-model outputs using the same prompts as the human evaluation. Its ratings remain relatively flat as imitation data increases, while larger base models receive stronger ratings.
- Evaluation procedure: GPT-4 ranks ChatGPT and imitation-model outputs presented in pairs using the same prompts as the human-preference evaluations.
- Results: As imitation data increases, GPT-4’s ratings of imitation-model outputs remain relatively flat.