Source-linked AI summary
Orca 2: Teaching Small Language Models How to Reason
Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, Hamid Palangi, Guoqing Zheng, Corby Rosset, Hamed Khanpour, Ahmed Awadallah
TL;DR
Small language models trained mainly by imitation may miss effective reasoning strategies and lack knowledge beyond pre-training. Orca 2 trains them on varied, task-tailored reasoning behaviors and teaches strategy selection, then evaluates them across 15 benchmarks. It significantly surpasses similarly sized models and matches or exceeds models 5 to 10 times larger, especially on zero-shot reasoning tasks.
Problem
Imitation-based training can leave smaller models short of teacher-level reasoning and comprehension, while instruction tuning does not necessarily teach new knowledge beyond pre-training.
Method
Orca 2 uses task-tailored synthetic demonstrations of multiple reasoning strategies and trains smaller models to select an effective strategy for each task.
Results
Orca 2 significantly surpasses similarly sized models and matches or exceeds models 5 to 10 times larger, especially on reasoning tasks, across 15 benchmarks covering approximately 100 tasks and over 36,000 prompts.
Takeaways & Limitations
Tailored synthetic data and strategy selection can improve smaller models’ reasoning to levels comparable to or exceeding much larger models on zero-shot reasoning tasks.
Takeaways & Limitations
The models remain constrained by their underlying base models and have not undergone RLHF training for safety.
Abstract
from arXiv · showhide
Orca 1 learns from rich signals, such as explanation traces, allowing it to outperform conventional instruction-tuned models on benchmarks like BigBench Hard and AGIEval. In Orca 2, we continue exploring how improved training signals can enhance smaller LMs' reasoning abilities. Research on training small LMs has often relied on imitation learning to replicate the output of more capable models. We contend that excessive emphasis on imitation may restrict the potential of smaller models. We seek to teach small LMs to employ different solution strategies for different tasks, potentially different from the one used by the larger model. For example, while larger models might provide a direct answer to a complex task, smaller models may not have the same capacity. In Orca 2, we teach the model various reasoning techniques (step-by-step, recall then generate, recall-reason-generate, direct answer, etc.). More crucially, we aim to help the model learn to determine the most effective solution strategy for each task. We evaluate Orca 2 using a comprehensive set of 15 diverse benchmarks (corresponding to approximately 100 tasks and over 36,000 unique prompts). Orca 2 significantly surpasses models of similar size and attains performance levels similar or better to those of models 5-10x larger, as assessed on complex tasks that test advanced reasoning abilities in zero-shot settings. make Orca 2 weights publicly available at aka.ms/orca-lm to support research on the development, evaluation, and alignment of smaller LMs
1 Introduction
Orca 2 addresses limits of imitation-based training by teaching smaller language models multiple reasoning strategies and how to select among them. It evaluates these capabilities broadly and reports strong zero-shot reasoning performance relative to both similar-sized and much larger models.
- Imitation learning can reproduce teacher style while falling short on reasoning and comprehension, potentially limiting smaller models’ solution strategies.
- Orca 2 teaches step-by-step, recall-then-generate, recall-reason-generate, extract-generate, and direct-answer techniques.
- Orca 2 also aims to teach smaller models when to select the most effective reasoning strategy for each task.
- 15 benchmarks cover approximately 100 tasks and over 36,000 unique prompts across language understanding, reasoning, mathematics, comprehension, summarization, groundedness, truthfulness, and toxic-content tasks.
- Orca 2 significantly surpasses similarly sized models and matches or exceeds models 5 to 10 times larger, especially on reasoning tasks.
2 Preliminaries
Instruction tuning trains models on input-output demonstrations and can improve instruction following, generation quality, and zero-shot reasoning. However, smaller instruction-tuned models may imitate teacher style without gaining proportional performance or new knowledge.
- Instruction tuning learns from natural-language task descriptions paired with demonstrations of desired behavior.
- Smaller student models trained from larger-model outputs often mimic teacher style but may not improve proportionally on knowledge-intensive or reasoning-intensive tasks.
- Instruction tuning teaches models how to solve tasks but does not necessarily provide new knowledge beyond pre-training.
- Smaller models with enhanced reasoning may be best used as reasoning engines over knowledge supplied in context or within narrower domains.
- Explanation tuning addresses terse, stylistically correct but wrong outputs by training students on richer reasoning signals elicited from capable language models.
3 Teaching Orca 2 to be a Cautious Reasoner
Orca 2 develops cautious reasoning by matching task-specific solution strategies to task requirements and student-model capacity. Prompt Erasing then removes the teacher’s strategy instructions during training so the student can learn the underlying strategy rather than merely imitate its framing.
- Response quality can vary substantially with the system-instruction strategy, and not every strategy-task combination is appropriate.
- GPT-4 produced different story-reordering answers under different instructions, with only the final strategy yielding the correct answer.
- Task-specific prompts can guide models toward direct answers, step-by-step reasoning, explain-then-answer, or other strategies.
- Orca 2’s cautious reasoning selects among direct-answer and slow-thinking strategies according to the task.
- Prompt Erasing replaces the teacher’s detailed system instruction with a generic one, encouraging the student to learn the underlying strategy and associated reasoning abilities.
4 Technical Details
Orca 2 combines FLAN-derived, few-shot, mathematical, and synthetic data in a progressive training pipeline based on LLaMA-2 checkpoints. Training uses standardized tokenization, packed sequences, and loss computed on teacher-generated response tokens.
- Dataset and training data: The Orca 2 dataset contains approximately 817K training instances and is combined progressively with FLAN annotations and Orca 1 data.
- Dataset and training data: Approximately 602K zero-shot queries were selected from 1,448 high-quality FLAN-v2 tasks for cautious-reasoning data construction.
- Dataset and training data: The data sources also include 55K few-shot samples, approximately 160K math problems, and 2,000 synthetic doctor-patient conversations.
- Training process: Progressive learning fine-tunes LLaMA-2-7B or LLaMA-2-13B on FLAN-v2, Orca 1 ChatGPT data, and then combined Orca 1 GPT-4 and Orca 2 data.
- Training process: Training uses LLaMA BPE tokenization, packs examples up to max_len = 4096 tokens, and computes loss only on teacher-generated response tokens.
5 Experimental Setup
Orca 2 is evaluated against instruction-tuned baselines across diverse capabilities using zero-shot prompts and a standardized answer-processing pipeline. The setup also reports constraints from greedy decoding, prompt sensitivity, and partially unknown pretraining data.
- Baselines: The study benchmarks Orca 2 alongside instruction-tuned LLaMA-2, WizardLM, Orca 1, and GPT models.The comparison includes LLaMA-2 models at 13B and 70B parameters, WizardLM at 13B and 70B, and Orca 1 at 13B.
- Benchmarks: Evaluations cover open-ended generation, summarization, safety, bias, reasoning, comprehension, grounding, and truthfulness in zero-shot settings.The benchmark suite includes standardized tests, reading comprehension, commonsense inference, long-range context, clinical-note generation, web-query answering, meeting summarization, toxic-content evaluation, and alignment tasks.
- Constraints: Evaluations use greedy decoding without sampling and exclude few-shot analysis, while some datasets are in-domain and base-model training data remain incompletely known.The authors note that larger models benefit more from few-shot settings and that prompt sensitivity and extraction coverage can affect comparisons with other studies.
- Prompts and decoding: Models are evaluated with empty system messages and simple prompts, adding answer-format guidance only where needed for reliable extraction.Formatting instructions can require outputs such as “###Final answer: {answer choice},” while a cautious system message is also considered.
- Answer parsing: Answers are parsed as multiple-choice, exact-match/span-extraction, or open-ended outputs, with identical parsing logic applied across models.The pipeline also reports format-OK, the percentage of samples from which an answer could be extracted, and acknowledges that parsing could be improved.
6 Evaluation Results
Across zero-shot evaluations, Orca 2 outperforms similarly sized models on reasoning, knowledge, language understanding, and text completion tasks, while showing narrower or conditional advantages in conversations, grounding, and safety-related evaluations.
- Reasoning: Orca-2-13B improves zero-shot reasoning performance by 47.54% over LLaMA-2-Chat-13B and 28.15% over WizardLM-13B, despite sharing their base model.It exceeds LLaMA-2-Chat-70B and is comparable to WizardLM-70B and ChatGPT; Orca-2-7B is better or comparable to LLaMA-2-Chat-70B across reasoning tasks.
- Knowledge and Language Understanding: Orca-2-13B surpasses both same-size baselines on each MMLU, ARC-Easy, and ARC-Challenge benchmark, with average improvements of 25.38% and 44.22%.On MMLU, Orca-2-13B scores 57.73%, similar to LLaMA2-Chat-70B at 58.54% and WizardLM-70 at 55.00%; Orca-2-7B surpasses both 70B baselines on ARC.
- Text Completion: 33.13% and 61.94% are Orca-2-13B’s relative improvements over LLaMA-2-Chat-13B and WizardLM-13B on HellaSwag.Both Orca-2-7B and Orca-2-13B outperform the 13B and 70B baselines on HellaSwag, although chat-model suitability for text completion is questioned.
- Multi-Turn Open Ended Conversations: Orca-2-13B performs comparably with other 13B models on MT-Bench, but its average second-turn score is lower than its first-turn score.The paper attributes the turn-level decline to the absence of conversations in training data and identifies multi-turn conversational ability as future work.
- Grounding: Orca-2-13B has the lowest hallucination rate among the evaluated Orca 2 variants and other 13B and 70B LLMs, reducing hallucinations by 76.92% and 61.71% versus two 13B baselines.The cautious system message consistently increases hallucination rates across the three studied tasks, with manual analysis suggesting extrapolated, context-unsupported content in summaries.
- Safety and Toxicity: ToxiGen results show that Orca-2-7B classifies toxic statements better than neutral statements, whereas Orca-2-13B does not show this problem in the experiment.The paper notes that this imbalance may cause erasure of non-problematic content related to specific identity groups.
7 Limitations
Orca 2 remains constrained by limitations inherited from its base models and training data, including bias, opacity, hallucination, misuse, distribution shift, system-message variance, and limited safety evaluation.
- Orca 2 may inherit biases from its source data and generate outputs that are potentially biased or unfair.
- Large language models can obscure the rationale behind outputs because their complexity makes them difficult to interpret.
- Without safeguards, Orca 2 could be misused to generate disinformation or harmful content.
- Orca 2’s performance may correlate strongly with tuning-data distribution, limiting accuracy in underrepresented areas such as math and coding.
- Performance varies with system instructions, and model-size stochasticity may produce nondeterministic responses to different instructions.
- Orca 2 was trained mainly for zero-shot settings and has not demonstrated the same few-shot gains as other, especially larger, models.
- Synthetic-data training may transmit both benefits and shortcomings from the data-generation models and methods, while detailed risk quantification remains necessary.
- Post-training improves task-solving and reasoning but does not necessarily add knowledge, leaving Orca 2 dependent on pre-training and better suited as a reasoning engine over supplied knowledge.
8 Conclusions
The study shows that smaller language models can improve reasoning through tailored synthetic-data training. Orca 2 combines multiple reasoning techniques with task-specific strategy selection and reaches performance comparable to or exceeding much larger models, especially in zero-shot reasoning.
- Orca 2 uses varied reasoning techniques and selects an effective strategy for each task, achieving performance comparable to or exceeding much larger models on zero-shot reasoning tasks.
A AGIEval Subtask Metrics
AGIEval results compare Orca 2 with same-size and larger baselines across individual subtasks and aggregate scores. Orca 2 generally performs competitively, while some task-specific deficits remain.
- AGIEval subtask performance is measured using exact-match accuracy under zero-shot evaluation.
- Orca-2-13B achieves average scores of 49.93% with an empty system message and 48.18% with a cautious system message.
- Orca-2-7B achieves average scores of 45.10% with an empty system message and 43.97% with a cautious system message.
- +11.08 points: Orca-2-13B outperforms LLaMA-2-Chat-13B on average, including gains of +22.31 points on LSAT-RC and +14.70 points on Gaokao EN.
- +11.68 points: Orca-2-13B surpasses WizardLM-13B on average, with advantages of +15.99 points on LSAT-RC and +12.74 points on Gaokao EN.
- +3.23 points: Orca-2-13B leads LLaMA-2-70B on average despite having around five times fewer parameters.
- Orca-2-13B outperforms Orca-1-13B on average by +4.24 points, while the cautious variant leads by +2.49 points.
- Orca 2 surpasses models as large as 70B parameters on zero-shot reasoning tasks, although individual AGIEval subtasks include shortfalls such as LSAT-LR and GAOKAO-EN.
B BigBench-Hard Subtask Metrics
The paper reports zero-shot evaluation across BBH reasoning tasks and examines hallucination rates in abstractive summarization. Tables organize task-level accuracy, while the hallucination analysis identifies Orca-2-13B as the strongest evaluated model.
- BBH task metrics: Accuracy is the evaluation metric for Orca 2 and baseline models across BBH multiple-choice reasoning tasks.Tables 7–10 cover all listed BBH task groups.
- BBH task metrics: The BBH results are distributed across four table groups covering Tasks 1–24.Tables 7, 8, 9, and 10 correspond to Tasks 1–6, 7–14, 15–20, and 21–24.
- Hallucination evaluation: Hallucination rate is judged by GPT-4, with lower values indicating better performance.Table 11 compares 13B and 7B Orca 2 variants with baseline models.
- Hallucination evaluation: Orca-2-13B emerges as the most effective model among Orca 2 versions and models of comparable size in hallucination-rate evaluation.The comparison is reported for the table’s upper and lower segments.
C.2 Evaluation of Generated Summaries
Generated summaries are evaluated with automatic overlap and language-model metrics, alongside comparisons across datasets, model variants, and system messages. Orca 2 performs competitively, but cautious system messages can reduce agreement with human reference summaries.
- Metrics: BLEU, ROUGE-L, and perplexity are used to evaluate abstractive-summary quality.For perplexity, lower values are better.
- Dataset results: Orca 2 outperforms both LLaMA-2-Chat variants on ACI-BENCH and is comparable to WizardLM-70B.On QMSum, Orca-2-13B and Orca-2-7B outperform LLaMA-2-Chat-70B and WizardLM-70B.
- Metric comparisons: Orca-2-13B performs better than other Orca 2 models on ACI-BENCH and QMSum under n-gram-based metrics.GPT-4 achieves the best ACI-BENCH performance among the listed baselines.
- System-message comparison: Direct system messages generally outperform cautious system messages across the evaluated summarization datasets.The paper links cautious-message outputs to greater rephrasing and extrapolation from the source.
D Evaluation of Safety
The safety evaluation section provides prompt materials and dataset-specific breakdowns for neutral and toxic statement classification. It describes the evaluation setup but supplies no outcome values in the cited passages.
- Evaluation materials: The appendix provides additional results for the experiments described in the safety evaluation section.The cited passage introduces further experimental details without reporting metrics.
- Evaluation materials: ToxiGen results are broken down by target identity group in the discriminative evaluation regime.These results are described as a breakdown of aggregated results from Section 6.6.
- Classification tasks: Tables 13 and 14 are labeled for neutral statement classification and toxic statement classification, respectively.The cited table labels identify the two classification evaluations.
- Prompt design: Evaluation prompts are simple and provide hints about answer format to improve parsing, while formatted questions are used as-is.Examples from all datasets appear in Appendix F.
F Illustrative Example from Evaluation Benchmarks and Corresponding Model Output
The appendix presents illustrative evaluation prompts and Orca 2 outputs spanning reasoning, knowledge, summarization, grounding, and safety-related tasks. Examples include both outputs without detected hallucinations and outputs containing judge-identified factual errors.
- Illustrative benchmark examples: The examples include math, reading comprehension, commonsense, history, logical reasoning, and experimental-design questions.Examples come from GSM8k, DROP, RACE, ARC, HellaSwag, LAMBADA, MMLU, and related evaluations.
- Illustrative benchmark examples: Several figures show Orca-2-13B responses generated with a cautious system message.The demonstrations cover SAT math, DROP, CRASS, RACE, BBH, GSM8k, MMLU, ARC, HellaSwag, and LAMBADA.
- Grounded summarization examples: The ACI-BENCH example contrasts a summary with no detected hallucination against a cautious-message output containing the unsupported term “knee joint.”The latter error is identified by a GPT-4 judge because the context mentions only “knee.”
- Grounded summarization examples: The QMSum example reports no hallucination for the direct output but two incorrect facts in the cautious output.The cited errors concern who goes first and an animal described as “liver.”
- Grounded summarization examples: The MS-MARCO example uses retrieved facts to answer a question and highlights context excerpts associated with a possible later hallucination.A corresponding Orca-2-13B output is reported with no detected hallucination.