PaperlayerView exact revision on arXiv

Source-linked AI summary

Orca: Progressive Learning from Complex Explanation Traces of GPT-4

Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, Ahmed Awadallah

arXiv:2306.02707v1cs.CLcs.LG

TL;DR

Smaller models trained to imitate large foundation models can match their style yet fall short in reasoning and comprehension. Orca applies Explanation Tuning, and it significantly outperforms other open-source smaller models, matching or surpassing ChatGPT in some settings while remaining substantially behind GPT-4.

  • Problem

    Smaller models imitating large foundation models can reproduce teacher style yet fall short in reasoning and comprehension, amid limitations in task diversity, query complexity, and data scaling.

    Source: §1 · p. 4, §1.1 · p. 5

  • Method

    Orca applies Explanation Tuning to align smaller models with GPT-4.

    Source: §9 · p. 29

  • Results

    Orca significantly outperforms other open-source smaller models and matches or surpasses ChatGPT in some settings, while a substantial gap with GPT-4 remains.

    Source: §9 · p. 29

  • Takeaways & Limitations

    The findings suggest learning from step-by-step explanations could improve model quality regardless of model size.

    Source: §9 · p. 29

    Whether smaller models are more susceptible to hallucination in ungrounded generation remains unclear.

    Source: §8 · p. 28

Abstract

from arXiv · show

Recent research has focused on enhancing the capability of smaller models through imitation learning, drawing on the outputs generated by large foundation models (LFMs). A number of issues impact the quality of these models, ranging from limited imitation signals from shallow LFM outputs; small scale homogeneous training data; and most notably a lack of rigorous evaluation resulting in overestimating the small model's capability as they tend to learn to imitate the style, but not the reasoning process of LFMs. To address these challenges, we develop Orca (We are working with our legal team to publicly release a diff of the model weights in accordance with LLaMA's release policy to be published at https://aka.ms/orca-lm), a 13-billion parameter model that learns to imitate the reasoning process of LFMs. Orca learns from rich signals from GPT-4 including explanation traces; step-by-step thought processes; and other complex instructions, guided by teacher assistance from ChatGPT. To promote this progressive learning, we tap into large-scale and diverse imitation data with judicious sampling and selection. Orca surpasses conventional state-of-the-art instruction-tuned models such as Vicuna-13B by more than 100% in complex zero-shot reasoning benchmarks like Big-Bench Hard (BBH) and 42% on AGIEval. Moreover, Orca reaches parity with ChatGPT on the BBH benchmark and shows competitive performance (4 pts gap with optimized system message) in professional and academic examinations like the SAT, LSAT, GRE, and GMAT, both in zero-shot settings without CoT; while trailing behind GPT-4. Our research indicates that learning from step-by-step explanations, whether these are generated by humans or more advanced AI models, is a promising direction to improve model capabilities and skills.

1 Introduction

The introduction argues that small instruction-tuned models can imitate foundation-model style without acquiring their reasoning skills. It presents explanation-driven training, diverse data, and more rigorous evaluation as Orca’s response.

1 Introduction

  • 92% of ChatGPT’s quality under GPT-4 judging fell to 64% on professional and academic exams and 48% on BigBench-hard for Vicuna, exposing evaluation overestimation.The introduction contrasts GPT-4-as-judge evaluation with assessments on reasoning benchmarks against human labels.Source: Figure 1 · p. 4, Figure 2 · p. 4, Figure 3 · p. 5
  • Models including Alpaca, WizardLM, and Vicuna can match teacher style yet often fall short in the reasoning and comprehension skills of larger foundation models.The introduction notes that larger models’ capabilities are associated with scaling and alignment training.Source: §1 · p. 4
  • Orca achieves parity with ChatGPT on BigBench-Hard and is described as having a 5 pts gap against ChatGPT across zero-shot GRE, GMAT, LSAT, and SAT exams.The exam comparison is reported without exemplars or chain-of-thought prompting.Source: Figure 2 · p. 4, Figure 3 · p. 5

1.1 Challenges with Existing Methods

  • Prior imitation tuning is limited in task diversity, query complexity, and scale, but the authors argue that large, diverse datasets can be assembled to reduce gaps on zero-shot reasoning benchmarks.They identify simple or repetitive instructions and limited data scale as shortcomings of existing approaches.Source: §1.1 · pp. 5–6
  • Teacher-generated query-response pairs provide limited reasoning signals, while richer LFM rationales can help close the gap when internal logits and representations are unavailable.Prior open-box distillation work uses logits, intermediate representations, and attention states; the cited closed-box alternative is rationales.Source: §1.1 · p. 6
  • Small test sets and GPT-4 judge biases—including preferences for longer answers and response order—can overestimate smaller models’ comprehension and reasoning abilities.The cited examples include 80 Vicuna instructions and 218 WizardLM instructions.Source: §1.1 · p. 6

1.2 Key Contributions

  • Explanation tuning augments query-response pairs with GPT-4 explanations elicited by system instructions, giving students additional signals to learn the teacher’s reasoning process.This contrasts with vanilla instruction tuning, which trains only on the prompt and LFM response.Source: §1.2 · p. 6
  • Orca samples diverse FLAN-v2 tasks and complex prompts, collects 5 million ChatGPT responses, then samples 1 million for GPT-4 responses to support progressive learning.The collection uses high-quality templates, formatting patterns, and data augmentations; ChatGPT serves as a teacher assistant.Source: §1.2 · p. 6

2 Preliminaries

The preliminaries define instruction tuning and summarize its use in language-only and multimodal tasks, then introduce system instructions as a way to elicit richer responses that expose GPT-4’s reasoning for smaller models to imitate.

2.1 Instruction Tuning

  • Instruction tuning trains pretrained language models on task descriptions paired with responses, and has improved FLAN’s and InstructGPT’s zero-shot and few-shot benchmark performance.It has also been used to generate synthetic instruction-following data for image captioning and visual question answering.Source: §2.1 · p. 7
  • Alpaca, Vicuna, WizardLM, and Koala use GPT-family outputs to instruction-tune smaller models, but the cited works are criticized for limited task diversity, query complexity, training scale, and evaluation.The passage says limited evaluation can overstate the benefits of this approach.Source: §2.1 · p. 7

2.2 Role of System Instructions

  • Vanilla instruction tuning’s short input-response pairs give smaller models limited ability to trace an LFM’s reasoning, while diverse system instructions can elicit structured outputs for them to imitate.The examples include chain-of-thought reasoning, “explain like I’m five,” and being helpful and informative; training uses system instruction, user instruction, and input paired with output.Source: §2.2 · pp. 7–8, Figure 5 · p. 8
  • System instructions can specify response tone, task, format, and limitations, and can also promote safety through constraints on harmful content, privacy, and uncertainty.They are written in natural language and separated from user messages by the system role in the JSON request.Source: §2.2 · p. 8

3 Explanation Tuning

Orca’s explanation tuning builds a diverse instruction dataset and progressively trains the model on teacher-generated responses, moving from ChatGPT to GPT-4. Its data and training design aim to elicit and learn detailed explanations and step-by-step reasoning.

3.1 Dataset Construction

  • The training data pairs FLAN-v2 queries with system messages and teacher responses: 5 million receive ChatGPT responses, and a sampled 1 million receive GPT-4 responses.The system message supplies context and guidelines, shapes response length, character, behavior, and format; the user query defines the task.Source: §3.1 · p. 8, §3.1.3 · p. 13

3.1.1 System Messages

  • Sixteen hand-crafted system messages elicit varied response lengths, formats, creative or informational answers, and explanations with step-by-step reasoning.The messages augment queries to guide the LFM’s response behavior.Source: §3.1.1 · p. 9

3.1.2 Dataset Description and Sampling from the FLAN-v2 Collection

  • For Orca training, only zero-shot queries are sampled and Dialogue is omitted; the 5-million set includes all approximately 150,000 queries from 18 Zero-Shot CoT tasks, plus 2.5 million Flan 2021 queries and 2 million T0 queries.Source: §3.1.2 · pp. 10, 12

3.1.3 ChatGPT as Teaching Assistant

  • GPT-4 responses were on average 1.5× longer than ChatGPT responses, enabling Orca to progress from shorter to more complex teacher explanations.Orca was trained first on FLAN-5M with ChatGPT augmentations and then on FLAN-1M with GPT-4 augmentations.Source: §3.1.3 · p. 13

3.2 Training

  • Training uses the LLaMA BPE tokenizer, packs examples into sequences capped at 2,048 tokens, and computes loss only on teacher-generated response tokens.The loss conditions response generation on the system message and task instructions.Source: §3.2 · pp. 13–14

4 Experiment Setup

Orca is evaluated across open-ended generation, reasoning, and comprehension using broad task sets and a standardized protocol. The setup compares Orca with several language-model baselines and reports GPT-4-judged quality across datasets.

4 Experiment Setup

  • The evaluation protocol covers writing, comprehension, analytical, mathematical, and logical reasoning abilities.Orca is compared with Text-Davinci-003, ChatGPT, GPT-4, and 13B Vicuna; ChatGPT and GPT-4 use the 2023-03-15-preview API, and Orca uses an empty system message and temperature 0.7 unless specified otherwise.Source: §4 · p. 14, §4.1 · p. 14

4.2 Tasks

  • The tasks span open-ended generation and complex reasoning and comprehension tasks.Dataset statistics for the evaluation benchmarks are presented in Table 5.Source: §4.2 · p. 15

4.2.1 Open-ended Generation Capabilities

  • GPT-4 rates open-ended responses from 0 to 10 using a Vicuna-based evaluation framework across three prompt collections.Vicuna prompts include 80 prompts across nine skills; Awesome ChatGPT prompts span 164 roles, while WizardLM prompts cover 29 skills with difficulty levels.Source: §4.2.1 · pp. 15–16

4.2.2 Reasoning Capabilities

  • Reasoning evaluation uses English multiple-choice subsets of AGIEval and 23-task BIG-Bench Hard in zero-shot settings without labeled examples or CoT.Responses are parsed using the AGIEval format: the first capital character is matched exactly to the gold answer ID, and models are penalized when the expected ID appears later.Source: §4.2.2 · p. 16
  • Orca retains 95% of ChatGPT quality and 85% of GPT-4 quality across datasets, a 10-point improvement over Vicuna, as assessed by GPT-4.Table 7 reports percentage improvement in overall score against the reference models; the Orca-over-Vicuna improvement is shown in parentheses.Source: Table 7 · p. 17

5 Evaluation for Open-ended Generation

With GPT-4 as judge, Orca retains most of ChatGPT’s and GPT-4’s quality across open-ended generation datasets, while evaluation notes a bias toward the first model in comparisons.

5 Evaluation for Open-ended Generation

  • 95% of ChatGPT quality and 85% of GPT-4 quality were retained by Orca across all datasets, with a 10-point aggregate improvement over Vicuna.Performance is reported as the candidate model’s percentage improvement in overall score relative to the reference model.Source: §5 · p. 17
  • 98% of ChatGPT quality and 89% of GPT-4 quality were retained by Orca on the Awesome prompts dataset spanning 164 open-ended generation roles.Orca also performed on par with ChatGPT on Vicuna’s original setting, using the Vicuna prompts dataset and GPT-4 as rater.Source: §5 · p. 17
  • GPT-4 evaluation showed positive bias toward the first model’s response; evaluations placed the reference model first, consistent with the Vicuna setup.The passage notes that this evaluator bias was also reported in.Source: §5 · p. 17

6 Evaluation for Reasoning

In zero-shot reasoning evaluations without exemplars or chain-of-thought, Orca substantially outperforms Vicuna on AGIEval and Big-Bench Hard, while performing near ChatGPT on BBH overall. Its performance varies by task, with strengths in some reasoning categories and weaknesses in others.

6.1 AGIEval Results

  • A GPT-4 rating of 84.87 for Vicuna-13B on WizardLM prompts indicates significant rating bias when Vicuna’s response is shown before ChatGPT’s.The cited comparison presents Vicuna as the candidate model and ChatGPT as the reference model, with response order specified in the prompt.Source: §6.1 · p. 17
  • On AGIEval, Orca is at par with Text-da-vinci-003, retains 88% of ChatGPT quality, trails ChatGPT by 5 points, significantly lags GPT-4, and outperforms Vicuna by 42%.Across tasks, ChatGPT beat Orca on 450 examples and Orca beat ChatGPT on 325; all models scored below human performance. Orca’s best system instruction left a 4.4-point gap against ChatGPT.Source: Table 8 · p. 18, §6.1 · p. 18, Table 9 · p. 19
  • Scaling explanation data by 5× with intermediate ChatGPT assistance improved aggregate AGIEval performance by 4.5 points over GPT-4-only augmentation.The comparison is between Orca trained on FLAN-5M ChatGPT plus FLAN-1M GPT-4 augmentations and Orca trained only on FLAN-1M GPT-4 augmentations.Source: §6.1 · p. 18, Table 10 · p. 19
  • AGIEval sample analysis found differences in domain knowledge, complex reasoning, context length, geometric reasoning, and LaTeX processing between ChatGPT-winning and Orca-winning examples.The respective ChatGPT-beats-Orca versus Orca-beats-ChatGPT shares were 15%/21% for domain knowledge, 14%/18% for complex reasoning, 16%/8% for long context, 2%/5% for geometric reasoning, and 2%/10% for LaTeX reasoning; the long-context split suggests an edge for ChatGPT.Source: §6.1 · pp. 19–20

6.2 Big-Bench Hard Results

  • On BBH, Orca excels over ChatGPT on entailment, semantic understanding, temporal and spatial reasoning, color reasoning, and causal judgment.It improves by 4.5% on formal fallacies, 11.1% on Disambiguation QA, 5.8% on Snarks, 102% on Temporal Sequences, 3.6% on Navigate, and 1.7% on Colored Objects; on causal judgment it is on par with GPT-4 and exceeds ChatGPT by 4.7%.Source: §6.2 · pp. 20–21
  • At 49.7%, Orca is marginally above ChatGPT on BBH overall, 113% above Vicuna, and 26% below GPT-4 in zero-shot evaluation without CoT.The authors note that GPT-4 has reported data contamination on BBH, while they are unaware of comparable issues with the cited training data for Orca or Vicuna.Source: §6.2 · p. 20, Table 11 · p. 21
  • Orca and ChatGPT reach parity on BBH translation-error detection, but the benchmark offers limited multilingual evaluation; reported BBH results use zero-shot prompting without exemplars or CoT.The authors note frequent updates to deployed OpenAI models and sensitivity to generation hyperparameters, and report checking prior results for replicability.Source: §6.2 · pp. 21, 23
  • On BBH, Orca is weaker than ChatGPT on world-knowledge, logical, geometric, and table-understanding tasks, though it performs better on movie recommendation.ChatGPT leads by at least 9% on Boolean expressions and Web of Lies, by at least 4.9% on three- and seven-object logical deduction, by 23% on geometric shapes, and by 7.4% on the penguins-in-a-table task; Orca does better on the five-object deduction task and movie recommendation.Source: §6.2 · p. 22

7 Evaluation for Safety

The paper reports limited safety evaluations of truthfulness and toxic-content generation, while emphasizing that these tests do not constitute a comprehensive safety assessment. It also discusses hallucination limits and tool-augmented models as a way to address gaps in factual knowledge.

7.1 Truthful Question Answering

  • TruthfulQA-MC19 evaluates 684 misconception-prone questions from 38 categories using a multiple-choice format with four answer choices.The authors prefer more comprehensive generation-based evaluation and note uncertainty about how much TruthfulQA data GPT-4 RLHF tuning included.Source: §7.1 · p. 23
  • Orca is significantly more truthful than Vicuna-13B but trails ChatGPT and GPT-4; it handles an ambiguous Sesame Street question correctly but gives an incorrect alcohol-purchase age.For the alcohol question, Orca answers 18, while ChatGPT and GPT-4 answer that the age depends on the country.Source: Figure 14 · p. 24, §7.1 · pp. 24, 26
  • Vicuna’s prompt-format failure rate is described as significantly higher than Orca’s, ChatGPT’s, and GPT-4’s.The paper notes that Vicuna’s outputs were often noisy and did not follow the prompt template, making them slower to label manually.Source: Figure 15 · p. 24, §7.1 · p. 24

7.2 Toxic Content Generation

  • The ToxiGen study covers a specific category of content harms and uses HateBERT to estimate toxicity, while acknowledging that detectors may be biased and human evaluation remains future work.A more comprehensive toxicity evaluation is left for future work; the paper also says that broader safety evaluation beyond the tested cases is crucial.Source: §7 · p. 23, §7.2 · p. 26
  • With toxic prompts, Orca is less prone to generate toxic content than Vicuna; with neutral prompts, it generates more neutral content than Vicuna and ChatGPT and almost as much as GPT-4.The comparisons use ToxiGen prompts; the figure captions attribute Orca’s relative safety to GPT-4 alignment and Azure OpenAI content-filter moderation during explanation tuning.Source: Figure 18 · p. 26, §7.2 · p. 26, Figure 19 · p. 27

7.3 Note on Hallucination and Tool Augmented LFMs

  • In a CV-generation case study, all models struggle with contact details, while GPT-4 produces more relevant professional summaries with fewer mistakes; the authors frame tool-augmented models as a way to retrieve knowledge.The paper says smaller models lose capacity as knowledge bases or memory stores but can remain reasoning engines, and cites tool-augmented work [34] [35], AutoGPT, Prometheus, and work offloading ChatGPT reasoning to 7B LLaMA.Source: §7.3 · pp. 27–28

8 Limitations

Orca retains common LLM limitations and has additional uncertainties tied to its explanation-tuning data, prompting conditions, and imitation of GPT-4. It is intended for research settings only, not downstream applications without further harm and bias analysis.

8 Limitations

  • Orca retains LLaMA-family and common LLM constraints, including source-data bias, weak real-world understanding, opaque rationales, content harms, and misuse risks.The paper recommends content-moderation services and calls for better regulation and standards, while recognizing research and open-source communities’ role.Source: §8 · p. 28
  • Orca may fabricate content, so it should not be fully relied on for critical decisions; whether smaller models hallucinate more in ungrounded generation remains unclear.The paper calls for more rigorous measurement, understanding, and mitigation of hallucination.Source: §8 · p. 28
  • Orca was tested only with simulated zero-shot standard prompts; multi-turn, in-context, few-shot, and advanced chain-of-thought prompting remain untested, and underrepresented data may limit accuracy.The paper specifically identifies math, coding, and reasoning as potentially underrepresented areas.Source: §8 · p. 28
  • Different system instructions can produce non-deterministic responses, which the authors associate with stochasticity from the model’s size.Source: §8 · p. 28
  • Imitating GPT-4 may pass along both its advantages and shortcomings; the authors posit safety benefits from GPT-4 training and Azure API guardrails, but say risks need further study.The paper notes that detailed studies are required to better quantify risks.Source: §8 · p. 28
  • Orca is designed and tested solely for research, not downstream applications without additional analysis of potential harm or bias in the proposed application.Source: §8 · p. 29

9 Conclusions

The conclusions emphasize Orca’s strong performance relative to smaller open-source models while noting a remaining gap with GPT-4. They also highlight robust evaluation, explanation-based imitation, and data and base-model quality as important to developing capable smaller models.

9 Conclusions

  • Smaller models’ abilities are frequently overstated, while standardized-test benchmarks such as AGIEval offer a more robust evaluation framework.The passage cites tests such as the GRE, SAT, and LSAT as examples.Source: §9 · p. 29
  • Explanation Tuning is effective for aligning smaller models to GPT-4, with data size and coverage and base-model quality also influencing performance.The authors note that more refined alignment methods remain needed.Source: §9 · p. 29
  • Orca significantly outperforms other open-source smaller models and can match or surpass ChatGPT in some settings, although a substantial gap with GPT-4 remains.The authors suggest smaller models can be more focused and adaptable in constrained settings without substantial loss in quality.Source: §9 · p. 29
  • Learning from step-by-step explanations generated by humans or more powerful AI models could improve model quality regardless of model size.Source: §9 · p. 29
  • Future work should develop more robust evaluation methods, advance alignment and post-training techniques, and use powerful models like GPT-4 more effectively as teachers.Source: §9 · p. 29

10 Author Contributions

The authors shared responsibility across project conception, data collection, training, evaluation, safety analysis, and manuscript writing. Their contributions ranged from leading implementation to technical advising and editing.

10 Author Contributions

  • Subhabrata Mukherjee conceived and led the project, drove implementation and experiments, designed data collection, training, and evaluation, and wrote major parts of the paper.He also helped collect data and train the model.Source: §10 · p. 29
  • Arindam Mitra contributed FLAN sampling and data collection, system-message design, training-pipeline setup, experiments, project brainstorming, and major parts of Sections 3 and 11.Source: §10 · p. 29
  • Ganesh Jawahar drove evaluation of open-ended generation and reasoning, ensured consistency with prior experimental setups, analyzed results, and wrote major parts of Sections 4, 5, and 6.Source: §10 · p. 29
  • Hamid Palangi led safety evaluation, designed and conducted TruthfulQA and Toxigen experiments, analyzed hallucination, participated in discussions, and wrote Section 7.Source: §10 · p. 29
  • Sahaj Agarwal helped collect data, participated in discussions, and contributed to an earlier version of the training-pipeline setup.Source: §10 · p. 29
  • Ahmed Awadallah advocated for the idea early, provided technical advice and feedback, participated in discussions, and substantially contributed to paper writing and editing.Source: §10 · p. 29

11 Case Studies · 11.1 Trigonometric Problem Solving · 11.2 Temporal Reasoning

The case studies compare Orca, Vicuna, ChatGPT, and GPT-4 responses under greedy decoding, using a right-triangle problem and an event-time reasoning question.

11 Case Studies

  • The paper compares responses from Orca, Vicuna, ChatGPT, and GPT-4, generating all responses with greedy decoding for reproducibility.Source: §11 · p. 30

11.1 Trigonometric Problem Solving

  • Orca also gives leg lengths 5 and 12 after setting the legs to 5x and 12x and solving for x.Its explanation incorrectly describes the hypotenuse as the sum of the squared leg lengths.Source: §11.1 · p. 30
  • Vicuna’s shown reasoning is erroneous: it states a^2 + b^2 = 26 and then derives a = 2b from the 5:12 ratio.The supplied passages do not show Vicuna’s final answer.Source: §11.1 · p. 30
  • ChatGPT correctly derives that the triangle’s legs are 5 and 12 using the Pythagorean theorem and the 5:12 ratio.Its response substitutes the ratio into the theorem before solving for both legs.Source: §11.1 · p. 31
  • GPT-4 sets up the triangle problem with a^2 + b^2 = 13^2 and the ratio a/b = 5/12, then substitutes a = (5/12)b.The supplied passage shows its setup and substitution, but not a final answer.Source: §11.1 · p. 31

11.2 Temporal Reasoning

  • Orca selects (B) 10am to 12pm as the only available time for Michael to visit the football field.The explanation excludes the other intervals because Michael was seen elsewhere during them.Source: §11.2 · p. 32

11.3 Multiple-choice Question-Answering · 11.4 Bio Olympiad · 11.5 Forming Inequalities

The examples show Orca answering a multiple-choice arithmetic question, reasoning through a rickets-related food question, and solving a profit inequality; the inequality responses vary in correctness.

11.3 Multiple-choice Question-Answering

  • In the apples question, Orca represents the amount as x = 20 - 10, obtains 10, and selects (E) None of the above.Source: §11.3 · p. 33

11.4 Bio Olympiad

  • The Bio Olympiad example identifies vitamin D deficiency as the cause of rickets and notes that none of the listed foods is a significant vitamin D source.Butter is described as the closest option, but not a significant source.Source: §11.4 · p. 34
  • For rickets, the response recommends fatty fish, egg yolks, fortified dairy products, and more sunlight exposure.Source: §11.4 · p. 34

11.5 Forming Inequalities

  • Orca’s profit calculation gives n > 70 as the condition for the company to make a profit.It compares income with cost and solves 12n > 7n + 350.Source: §11.5 · p. 36
  • ChatGPT and GPT-4 also derive n > 70 from the condition 12n > 7n + 350.ChatGPT checks the result with n = 80, where income is $960 and cost is $910.Source: §11.5 · p. 37

11.6 Counterfactual Question Answering … 11.9 Spatial Reasoning

The supplied case studies show Orca answering compound-growth, lifetime-blink, and spatial-reasoning questions; the counterfactual section is represented only by a figure caption.

11.6 Counterfactual Question Answering

  • Figure 25 is labeled as a counterfactual-question case study, but the supplied passage contains no question or answer.The figure caption identifies the case-study topic only.Source: Figure 25 · p. 38

11.7 Compound Interest Problems

  • Orca concludes that an item growing 20% annually takes 5 years to double, applying a formula for doubling time.The supplied passages introduce the formula and then plug in the 20% growth rate, but do not show the formula itself.Source: §11.7 · p. 38

11.8 Question from Vicuna-Eval

  • 64,800 blinks is Orca’s selected lower lifetime estimate, versus its stated upper estimate of 86,400 blinks.Orca chooses the lower estimate from calculations using 15 and 20 blinks per minute, respectively.Source: §11.8 · p. 40
  • Vicuna estimates 120,000,000–160,000,000 lifetime blinks, while ChatGPT calculates 840,960,000 blinks, or approximately 840 million.Vicuna’s estimate assumes an 80-year lifespan; ChatGPT also multiplies its yearly estimate by 80 years.Source: §11.8 · p. 40
  • Orca notes that the actual lifetime blink count may vary with age, health, and medication use because blink rate can change.This caveat qualifies its estimate, which assumes a constant blink rate across a lifetime.Source: §11.8 · p. 40

11.9 Spatial Reasoning

  • Orca concludes that A is immediately to the right of P in the seating arrangement.The supplied answer states this conclusion after considering the positions of S, Z, A, P, and R.Source: §11.9 · p. 41

11.10 Commonsense Question Answering … 11.13 Meeting Transcript Processing

These case studies show Orca producing step-by-step responses across commonsense questions, a CV prompt, equation solving, and transcript analysis, with outputs varying in accuracy, format, and supporting evidence.

11.10 Commonsense Question Answering

  • Orca gives 30 hours for drying 30 clothes, while Vicuna gives 150 hours, illustrating their different proportional reasoning on the same question.Orca cross-multiplies 5 clothes / 5 hours = 30 clothes / x hours; Vicuna multiplies 5 hours by 30 clothes.Source: §11.10 · p. 42
  • For measuring 6 liters, Orca describes filling the 12-liter jug and transferring liquid to the 6-liter jug, while GPT-4 fills the 6-liter jug and pours it into the 12-liter jug.Orca’s described sequence is more elaborate; GPT-4’s two steps leave 6 liters in the larger jug.Source: §11.10 · p. 43

11.11 Hallucination

  • The CV case shows GPT-4 adding polished career claims, while ChatGPT supplies a different University of Washington address in the generated profile.The GPT-4 text describes research contributions, teaching, and awards; ChatGPT lists the Department of Psychology address.Source: §11.11 · p. 44, Figure 31 · p. 45

11.12 Quadratic Equation Solving

  • Orca factors x2 + 3.0x = 0 as x(x + 3) = 0 and gives roots [-3, 0], but its response is marked correct answer, wrong format.The prompt requested the result in the form ROOTS = [root1, root2...].Source: §11.12 · p. 45

11.13 Meeting Transcript Processing

  • Orca extracts Steven’s statements and characterizes him as cautiously positive about selling, noting both potential financial and growth benefits and concerns about culture and control.The transcript shows Steven first supporting the opportunity, then sharing John’s concern that the company’s culture could change.Source: §11.13 · p. 47
  • Vicuna calls Steven hesitant about selling but provides no supporting transcript segments, unlike Orca’s explicit extraction-and-analysis response.The comparison is presented in the transcript case study; the supplied Vicuna passage describes its answer but is truncated.Source: §11.13 · p. 47

Related papers

Loading 2306.02707v1…