Source-linked AI summary
Orca: Progressive Learning from Complex Explanation Traces of GPT-4
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, Ahmed Awadallah
TL;DR
Smaller models trained from large-model outputs may imitate style without reproducing reasoning, and existing data and evaluation practices limit reliable assessment. Orca addresses this with GPT-4 explanation traces, progressive ChatGPT-to-GPT-4 supervision, and diverse sampled tasks. It outperforms open-source smaller models and can match or surpass ChatGPT in some settings, while a substantial gap with GPT-4 remains.
Problem
Existing smaller-model imitation can reproduce large models’ style without their reasoning, amid limited task diversity, data scaling, and rigorous evaluation.
Method
Orca uses diverse sampled imitation data augmented with GPT-4 explanations and step-by-step reasoning, with ChatGPT serving as an intermediate teacher.
Results
Orca significantly outperforms other open-source smaller models and, in some settings, matches or surpasses ChatGPT, while remaining behind GPT-4.
Takeaways & Limitations
Learning from step-by-step explanations could improve model quality across model sizes, while standardized benchmarks provide more robust evaluation of smaller models.
Takeaways & Limitations
Orca retains LLaMA-family constraints and common large-language-model limitations, including unresolved risks of hallucinated content.
Abstract
from arXiv · showhide
Recent research has focused on enhancing the capability of smaller models through imitation learning, drawing on the outputs generated by large foundation models (LFMs). A number of issues impact the quality of these models, ranging from limited imitation signals from shallow LFM outputs; small scale homogeneous training data; and most notably a lack of rigorous evaluation resulting in overestimating the small model's capability as they tend to learn to imitate the style, but not the reasoning process of LFMs. To address these challenges, we develop Orca (We are working with our legal team to publicly release a diff of the model weights in accordance with LLaMA's release policy to be published at https://aka.ms/orca-lm), a 13-billion parameter model that learns to imitate the reasoning process of LFMs. Orca learns from rich signals from GPT-4 including explanation traces; step-by-step thought processes; and other complex instructions, guided by teacher assistance from ChatGPT. To promote this progressive learning, we tap into large-scale and diverse imitation data with judicious sampling and selection. Orca surpasses conventional state-of-the-art instruction-tuned models such as Vicuna-13B by more than 100% in complex zero-shot reasoning benchmarks like Big-Bench Hard (BBH) and 42% on AGIEval. Moreover, Orca reaches parity with ChatGPT on the BBH benchmark and shows competitive performance (4 pts gap with optimized system message) in professional and academic examinations like the SAT, LSAT, GRE, and GMAT, both in zero-shot settings without CoT; while trailing behind GPT-4. Our research indicates that learning from step-by-step explanations, whether these are generated by humans or more advanced AI models, is a promising direction to improve model capabilities and skills.
1 Introduction
Smaller models trained on large-model outputs can imitate style without matching reasoning, while limited data diversity, imitation signals, and evaluation protocols obscure this gap. Orca addresses these issues with explanation tuning, diverse sampled tasks, and broader evaluation across reasoning, academic, professional, safety, and generation settings.
- Teacher-generated imitation data can match large models’ style while falling short in reasoning and comprehension.
- 92% versus 64% and 48%: Vicuna’s apparent ChatGPT quality varies sharply between GPT-4 judging and human-labeled reasoning benchmarks.
- Existing instruction-tuning data is constrained by simple, homogeneous instructions, limited task diversity, and small-scale collection.
- Query-response imitation provides limited signals for tracing teacher reasoning, whereas rationales offer richer supervision for task-specific distillation.
- Orca augments query-response pairs with GPT-4 reasoning explanations, samples diverse FLAN-v2 tasks, and evaluates multiple capabilities and benchmark types.
2 Preliminaries
Prior instruction-tuning approaches train smaller models on input-response pairs, but short responses provide limited access to the teacher’s reasoning. Orca instead uses system instructions to elicit richer, structured responses for explanation tuning.
- Instruction tuning trains language models from natural-language task descriptions paired with responses.
- Existing smaller-model approaches often use short, terse responses that provide limited ability to trace a large model’s reasoning process.
- Orca uses system instructions including chain-of-thought and step-by-step prompts to train on richer responses that mimic GPT-4’s thinking process.
- System instructions specify response behavior such as tone, format, task constraints, and safety requirements.
3 Explanation Tuning
Orca constructs explanation-tuning data by sampling diverse FLAN-v2 tasks, augmenting queries with varied system messages, and collecting responses from progressively stronger teachers. ChatGPT supervision precedes GPT-4 supervision, exposing the student to increasing explanation complexity.
- Dataset sampling: The training data selectively samples diverse task mixtures from FLAN-v2 rather than using the entire collection uniformly.
- Data construction: Orca’s training instances contain a system message, user query, and large-foundation-model response.
- System messages: Sixteen handcrafted system messages elicit varied answer lengths, formats, creative outputs, information-seeking responses, explanations, and step-by-step reasoning.
- Dataset sampling: The dataset combines collections including CoT, NiV2, T0, and Flan 2021, with zero-shot queries sampled across tasks.
- Teacher augmentation: 5 million instructions form FLAN-5M, while 1 million are separately used for GPT-4 responses in FLAN-1M.
- Progressive learning: Orca trains first on ChatGPT augmentations and then on GPT-4 augmentations as progressive learning.
- Progressive learning: 1.5×: GPT-4 responses are longer on average than ChatGPT responses, supporting progressively more complex teacher explanations.
4 Experiment Setup
The experiments evaluate Orca across generative, reasoning, comprehension, analytical, mathematical, logical, and safety capabilities. They combine GPT-4 judging with academic and professional benchmarks, standardized response parsing, and comparisons against multiple foundation-model baselines.
- Evaluation scope: The evaluation protocol covers writing, comprehension, analytical, mathematical, and logical reasoning abilities.
- Baselines: Orca is compared with Text-Davinci-003 and ChatGPT, alongside other evaluation baselines.
- Generative evaluation: GPT-4 rates model generations from 0 to 10 using Vicuna, Awesome, and WizardLM prompt collections.
- Reasoning benchmarks: AGIEval uses English multiple-choice questions from professional and academic exams, while Big-Bench Hard contains 23 challenging tasks.
- Evaluation protocol: Reasoning evaluations use zero-shot prompts without exemplars or chain-of-thought, with consistent answer parsing across models.
- Reported comparison: GPT-4 judging reports Orca at 95% of ChatGPT quality and 85% of GPT-4 quality across aggregated datasets.
5 Evaluation for Open-ended Generation
Orca performs strongly in GPT-4-judged open-ended generation evaluations, retaining much of ChatGPT’s quality and improving substantially over Vicuna, though the evaluation has documented judge-order bias.
- Orca retains 95% of ChatGPT quality and 85% of GPT-4 quality across the evaluated datasets.It shows a 10-point aggregate improvement over Vicuna.
- Orca performs on par with ChatGPT on Vicuna’s original prompt-based evaluation setting.
- Orca retains 98% of ChatGPT quality on the Awesome prompts dataset spanning 164 open-ended generation roles.
- GPT-4 evaluation shows positive bias toward the first response in a comparison, limiting interpretation of these judged results.The authors report this bias as also observed in prior work.
6 Evaluation for Reasoning
Across zero-shot reasoning evaluations without chain-of-thought, Orca substantially outperforms Vicuna and approaches ChatGPT on aggregate, while remaining behind GPT-4 and showing uneven task-level performance.
- AGIEval Results: 42% relative improvement over Vicuna is reported for Orca on AGIEval across categories.Orca performs at par with Text-Davinci-003 and retains 88% of ChatGPT quality, while significantly lagging GPT-4.
- Scaling and Teacher Assistance: Scaling explanation data fivefold with intermediate ChatGPT assistance improves aggregate performance by 4.5 points.The comparison uses GPT-4-only augmentation versus the full ChatGPT-and-GPT-4 training setup.
- AGIEval Results: 4.4 points is Orca’s performance gap against ChatGPT when the best system instruction is selected for each AGIEval task.
- Big-Bench Hard Results: 113% is Orca’s improvement over Vicuna on Big-Bench Hard under standard zero-shot prompting without exemplars or chain-of-thought.Orca performs marginally better than ChatGPT on aggregate but significantly lags GPT-4.
- Big-Bench Hard Results: 49.7% is Orca’s average Big-Bench Hard performance, placing it 26% behind GPT-4.The paper notes reported GPT-4 data contamination concerns for Big-Bench.
- Big-Bench Hard Results: 11.1% and 5.8% are Orca’s improvements over ChatGPT on Disambiguation QA and Snarks, respectively.
- Big-Bench Hard Results: Orca underperforms ChatGPT on world-knowledge tasks such as sports and artist-related humor, but performs better on movie recommendation.
- Big-Bench Hard Results: 23% is ChatGPT’s advantage over Orca on the geometric shape task, while ChatGPT also leads by at least 9% on Boolean expressions and Web of lies.
7 Evaluation for Safety
The safety evaluation finds Orca more truthful and less toxic than Vicuna in the tested settings, with neutral-content behavior near GPT-4, but the assessment covers only limited safety dimensions.
- Scope Limitation: A more comprehensive safety evaluation across dimensions not covered here is identified as future work.
- Truthfulness: Orca’s outputs are significantly more truthful than Vicuna-13B’s while trailing ChatGPT and GPT-4 on TruthfulQA.
- Instruction Following: Orca follows requested output formats more reliably than Vicuna, whose outputs are generally noisier and require more manual labeling.
- ToxiGen: The ToxiGen experiment uses HateBERT toxicity probabilities on outputs prompted with toxic and benign examples across 13 categories.The authors caution that off-the-shelf toxicity detectors have their own biases and weaknesses.
- ToxiGen: Orca generates less toxic content than Vicuna from toxic prompts and more neutral content from neutral prompts.
- ToxiGen: Orca generates almost equivalently neutral content as GPT-4 in the reported ToxiGen comparison.
8 Limitations
Orca inherits common large-language-model limitations involving bias, contextual understanding, transparency, misuse, hallucination, and safety. Its performance and applicability are also constrained by tuning-data distribution, system-instruction variability, and untested deployment contexts.
- Orca may generate biased or unfair outputs because large language models can carry biases from source data.
- Limited real-world understanding can produce inaccurate or nonsensical responses.
- Large language models also present risks involving limited transparency, content harms, hallucination, and potential misuse without safeguards.
- Performance remains untested in multi-turn, in-context, few-shot, and advanced chain-of-thought prompting contexts.
- Orca’s performance may correlate with tuning-data distribution, limiting accuracy in underrepresented areas such as math, coding, and reasoning.
- Orca may produce non-deterministic responses to different system instructions because of stochasticity associated with model size.
- The model may inherit GPT-4’s advantages and shortcomings, while risks require further quantification and testing beyond research settings.
9 Conclusions
The paper argues that smaller models are often overestimated and that robust benchmarks provide stronger evaluation. It presents explanation tuning, data coverage, and base-model quality as important factors, while Orca outperforms other smaller open-source models and can approach ChatGPT in some settings but remains behind GPT-4.
- Smaller models’ abilities are frequently overstated, while benchmarks such as AGIEval provide more robust evaluation frameworks.
- Explanation Tuning is presented as an effective method for aligning smaller models to GPT-4.
- Data size, data coverage, and base-model quality are identified as important factors influencing alignment and performance.
- Orca significantly outperforms other open-source smaller models and can match or surpass ChatGPT in some settings, while a substantial gap with GPT-4 remains.
- The findings support learning from step-by-step explanations as a way to improve model quality across model sizes.
- The authors hope the findings inform more robust evaluation, improved alignment and post-training, and more effective use of GPT-4 as a teacher.
10 Author Contributions
The author-contributions section identifies responsibilities spanning project conception, data collection, training, evaluation, system-message design, implementation, and experiments.
- Subhabrata Mukherjee proposed and led the project, designed data collection, training, and evaluation, and wrote major parts of the paper.
- Arindam Mitra contributed to sampling and data collection, system-message design, the training pipeline, and experiments.
11 Case Studies
The case studies compare Orca, Vicuna, ChatGPT, and GPT-4 on trigonometric problem solving and temporal reasoning. They illustrate both step-by-step solution traces and differences in answer correctness and reasoning quality.
- Case studies: The case-study comparisons use responses from Orca, Vicuna, ChatGPT, and GPT-4 generated with greedy decoding.
- Trigonometric problem solving: The trigonometry problem gives a right triangle with hypotenuse 13 and leg ratio 5:12.
- Trigonometric problem solving: Orca also concludes that the two leg lengths are 5 and 12 after reasoning with scaled legs 5x and 12x.
- Trigonometric problem solving: Vicuna’s displayed reasoning contains an incorrect Pythagorean substitution before using the leg ratio.
- Trigonometric problem solving: ChatGPT concludes that the two leg lengths are 5 and 12.
- Trigonometric problem solving: GPT-4 sets up the Pythagorean relationship and the 5:12 leg ratio before substituting one variable into the other equation.
- Trigonometric problem solving: GPT-4’s solution proceeds by solving for one leg in terms of the other and substituting into the Pythagorean equation.
- Temporal reasoning: The temporal-reasoning case asks when Michael could have visited the football field given wake-up, observation, and closing-time constraints.
11.3 Multiple-choice Question-Answering
The case studies compare Orca with other models on multiple reasoning tasks, including answer selection and profit inequalities. Orca provides explicit intermediate reasoning and reaches the stated solution in the highlighted profit example.
- Orca explains the rickets question by identifying vitamin D deficiency and comparing the food options.It identifies butter as the closest listed source while recommending vitamin D-rich foods such as fatty fish, egg yolks, and fortified dairy products.
- Orca solves the profit condition as n > 70 by comparing income 12n with cost 7n + 350.
- Vicuna instead reports that profitable production begins at n ≥ 39 after an incorrect simplification of the inequality.
11.7 Compound Interest Problems
The case studies examine compound-growth estimation, commonsense numerical reasoning, spatial reasoning, and answer interpretation. The examples show both step-by-step responses and substantial variation among model answers.
- Compound Interest Problems: Orca estimates that 20% annual growth doubles an item’s value in 5 years.
- Compound Interest Problems: Vicuna applies d = 70 / r and estimates 3.5 years for doubling at 20% growth.
- Spatial Reasoning: For the seating problem, Orca answers that A is to the right of P, whereas ChatGPT says the answer cannot be determined.
11.10 Commonsense Question Answering
These case studies test proportional reasoning, jug manipulation, factual generation, algebraic solving, and transcript understanding. The examples include both detailed reasoning and errors in answers or requested formats.
- Jug Measurement: The jug example shows that filling the 12-liter jug and transferring water into the 6-liter jug leaves exactly 6 liters.
- Quadratic Equation Solving: In the quadratic example, Orca factors x² + 3.0x = 0 as x(x + 3) = 0 and reports roots [-3, 0].The case study labels the response as correct but in the wrong format.
- Meeting Transcript Processing: The transcript case contrasts Steven’s support for selling’s benefits with his concern about cultural and product changes after acquisition.
- Meeting Transcript Processing: For transcript understanding, Orca characterizes Steven as cautiously positive while noting concerns about company culture and business control.