Source-linked AI summary
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, Jason Wei
TL;DR
The paper asks whether the 23 BIG-Bench tasks where prior models fell below average human-rater performance are truly beyond current language models. It identifies BIG-Bench Hard, evaluates chain-of-thought prompting and its interaction with model scale, and finds that CoT substantially improves performance, with Codex surpassing the human-rater baseline on 17 of 23 tasks.
Problem
The paper examines which BIG-Bench tasks remain below average human-rater performance and whether those tasks are unsolvable by current language models.
Method
The authors curate 23 challenging tasks into BIG-Bench Hard, manually create chain-of-thought exemplars, and evaluate answer-only versus CoT prompting across models and scales.
Results
Codex with CoT surpasses the average human-rater baseline on 17 of 23 BBH tasks, while CoT also improves performance across model families and can unlock performance on tasks with otherwise flat scaling curves.
Takeaways & Limitations
Answer-only few-shot evaluation underestimates language-model capabilities on challenging multi-step tasks, which are better captured with CoT prompting.
Takeaways & Limitations
The reported human-rater baselines are not fully representative of human performance because task formatting changed, evaluation was subsampled, external resources were allowed, and some descriptions were difficult to follow.
Abstract
from arXiv · showhide
BIG-Bench (Srivastava et al., 2022) is a diverse evaluation suite that focuses on tasks believed to be beyond the capabilities of current language models. Language models have already made good progress on this benchmark, with the best model in the BIG-Bench paper outperforming average reported human-rater results on 65% of the BIG-Bench tasks via few-shot prompting. But on what tasks do language models fall short of average human-rater performance, and are those tasks actually unsolvable by current language models? In this work, we focus on a suite of 23 challenging BIG-Bench tasks which we call BIG-Bench Hard (BBH). These are the task for which prior language model evaluations did not outperform the average human-rater. We find that applying chain-of-thought (CoT) prompting to BBH tasks enables PaLM to surpass the average human-rater performance on 10 of the 23 tasks, and Codex (code-davinci-002) to surpass the average human-rater performance on 17 of the 23 tasks. Since many tasks in BBH require multi-step reasoning, few-shot prompting without CoT, as done in the BIG-Bench evaluations (Srivastava et al., 2022), substantially underestimates the best performance and capabilities of language models, which is better captured via CoT prompting. As further analysis, we explore the interaction between CoT and model scale on BBH, finding that CoT enables emergent task performance on several BBH tasks with otherwise flat scaling curves.
1 Introduction
The paper identifies 23 BIG-Bench tasks where prior models did not exceed average human-rater performance and tests whether chain-of-thought prompting can improve results. It also examines how CoT interacts with model scale.
- BIG-Bench is a diverse evaluation suite designed to measure language-model capabilities and limitations across crowd-sourced tasks.
- The authors curate 23 particularly challenging tasks into the publicly available BIG-Bench Hard (BBH) suite.
- 17 of 23 BBH tasks exceed average human-rater performance when code-davinci-002 uses chain-of-thought prompting.
- The study also finds that CoT performance gains emerge with sufficient model size and can produce better-than-random performance on tasks with flat scaling curves.
2 BIG-Bench Hard
The authors filter BIG-Bench to create a clean, challenging, tractable benchmark of 23 tasks that prior results did not solve above average human-rater performance. BBH contains 6,511 evaluation examples across 23 tasks, including subtasks.
- BIG-Bench contains over 200 diverse text-based tasks spanning traditional NLP, mathematics, commonsense reasoning, and question-answering.
- Human-rater baselines were established by manually solving tasks and scoring responses against golden labels, although they are not representative of the entire population.
- 78 tasks remain after filtering for clean multiple-choice or exact-match evaluation; prior models exceeded average human-rater scores on 42, while 36 did not.
- 13 of the 36 remaining tasks are excluded because they require domain-specific knowledge, exceed practical solving time, or have ambiguous formulations.
- 23 tasks form BBH, including three subtasks each for Logical Deduction and Tracking Shuffled Objects, with 6,511 evaluation examples overall.
3 Experimental Setup
The study compares answer-only and chain-of-thought few-shot prompting across multiple model families and scales. CoT exemplars explicitly provide intermediate reasoning steps for BBH tasks.
- The experiments evaluate answer-only and chain-of-thought few-shot prompting across multiple scales of OpenAI models and PaLM.
- Few-shot prompting prepends input-output exemplars, task instructions, and answer options before the inference-time example.
- For each BBH task, the authors manually compose three CoT exemplars containing intermediate reasoning steps and prepend “let’s think step-by-step.”
- The evaluated model families are Codex, InstructGPT, and PaLM, with PaLM sizes of 8B, 62B, and 540B.
- Table 2 summarizes answer-only and CoT performance for InstructGPT text-davinci-002, Codex code-davinci-002, and PaLM 540B.
- Accuracy uses exact match after greedy decoding, with final answers extracted from expected keywords.
4 Results
CoT prompting improves BBH performance, especially as model scale increases, but its benefits vary across tasks and do not eliminate the gap with expert human-raters.
- 4.1 Standard answer-only prompting underestimates model capabilities: 6 of 23 tasks: PaLM 540B with answer-only prompting surpasses the average human-rater, and its result is 1.4% above the BIG-Bench reported result.The comparison uses the evaluation setup in this paper, which includes instructions and answer options in the prompt.
- 4.1 Standard answer-only prompting underestimates model capabilities: 17 of 23 tasks: Codex with CoT outperforms the average human-rater score, compared with 5 of 23 using answer-only prompting.CoT also improves Codex performance by more than 6% over the average human-rater, while remaining over 20% below the best human-rater performance.
- 4.2 Positive delta from chain-of-thought requires sufficient model scale: CoT’s performance delta over answer-only prompting increases with scale for OpenAI models and improves with larger PaLM models.For OpenAI models, CoT has negative or zero gains from text-ada-001 to text-curie-002, while the delta increases up to the largest model size; PaLM shows negative gain at 8B but improves at larger sizes.
- 4.3 Chain-of-thought unlocks emergent task performance: CoT enables emergent performance on Multi-Step Arithmetic, Tracking Shuffled Objects, and Web of Lies, whose answer-only scaling curves are flat.In these tasks, performance transitions from approximately random to improved performance as scale increases.
- 4.3 Chain-of-thought unlocks emergent task performance: CoT does not unlock emergent performance on every BBH task with a flat answer-only scaling curve.Causal Judgement reaches 57.8% with answer-only prompting and does not improve with CoT, compared with 50% random performance and a 100% highest human-rater score.
5 Discussion
The discussion organizes BBH findings by reasoning type, language understanding, world knowledge, and multilingual reasoning, while examining CoT gains and failures. CoT improves several algorithmic tasks but does not uniformly help tasks requiring world knowledge or sarcasm detection.
- Algorithmic and Multi-Step Arithmetic Reasoning: 60.4% ↑, 46.4% ↑, 46.0% ↑, and 19.8% ↑ are Codex’s CoT improvements on Tracking Shuffled Objects, Multi-Step Arithmetic, Navigate, and Temporal Sequences.These tasks span algorithmic and multi-step reasoning, where CoT can decompose complex problems into smaller solvable problems in sufficiently large models.
- Natural Language Understanding: PaLM and InstructGPT typically outperform Codex on natural-language understanding tasks, suggesting mostly natural-language training may suit these tasks better.The comparison covers disambiguation, adjective ordering, translation-error detection, and sarcasm detection.
- Use of World Knowledge: CoT helps Codex on Sports Understanding and Movie Recommendation, but Causal Judgment and Ruin Names worsen with CoT and remain below average human-rater performance.These tasks require factual or general world knowledge, including causal reasoning, humor, and cultural knowledge.
- Multilingual Knowledge and Reasoning: CoT improves Salient Translation Error Detection only for PaLM.The task concerns translation quality estimation and cross-lingual natural-language inference.
- Failure analysis of CoT Prompting: CoT gains 10–20% accuracy on average across three model families but lags answer-only prompting on Causal Judgment, Ruin Names, and Snarks.The Snarks examples indicate that sarcasm can be difficult to detect without situational information or underlying context.
- Human-Rater Baseline Performance: Human-rater scores are not fully representative because evaluation conditions, task formulations, sampling, time allocation, and external-resource access varied.The authors therefore caution that exceeding average human-rater performance should not be equated with true language understanding or reasoning.
6 Related Work
The related work situates BBH at the intersection of prompting, reasoning via prompting, and scaling and emergence. The paper’s distinctive claim is that CoT can unlock emergent task capabilities at sufficiently large model scales.
- Prompting: Prompting research established few-shot capabilities and developed methods for eliciting or analyzing language-model behavior.The cited work includes few-shot prompting, alternative prompting methods, and analyses of why prompting works.
- Scaling and emergence: Figure 7 provides examples where CoT prompting lags behind answer-only prompting on the Snark task.The figure is cited in the paper’s discussion of CoT failure modes.
- Reasoning via prompting: Reasoning-via-prompting research uses natural-language reasoning paths before or after answers for reasoning and explainability.The passage distinguishes reasoning paths used to improve explainability from the paper’s broader evaluation of reasoning performance.
- Scaling and emergence: CoT is presented as a key to unlocking emergent task capabilities in BBH at sufficiently large model scales.This connects scaling studies of model capacity and training data with research on capabilities that appear only at large scales.
7 Conclusion
The conclusion presents BBH as a 23-task challenge set for evaluating difficult language-model capabilities. It reports that CoT exposes stronger performance than answer-only prompting, especially for Codex.
- 7 Conclusion: CoT prompting shows that answer-only prompting underestimates model capabilities on BBH.The authors release the data, prompts, and Codex outputs for further research.
- 7 Conclusion: 17 out of 23 tasks exceed the average human-rater baseline with CoT prompting using the most capable Codex model.BBH consists of tasks where prior language models fell short of average reported human-rater performance.
A BIG-Bench Hard Task Descriptions
BBH includes diverse tasks spanning logical, mathematical, spatial, temporal, linguistic, factual, and commonsense reasoning. Each task specifies a structured prediction or classification problem.
- Logical and mathematical reasoning: Boolean Expressions and Web of Lies require evaluating Boolean truth values from symbolic or natural-language formulations.Boolean Expressions uses constants and basic operators, while Web of Lies expresses a Boolean function as a word problem.
- Natural-language understanding: Disambiguation QA resolves ambiguous pronouns or labels a sentence inherently ambiguous when no antecedent can be inferred.The task’s output is either an ambiguity judgment or the pronoun’s antecedent.
- Structured and spatial reasoning: Dyck Languages predicts missing closing parentheses, and Geometric Shapes identifies the shape generated by a complete SVG path.These tasks require structured sequence completion or geometric interpretation.
- Logical and mathematical reasoning: Formal Fallacies Syllogisms Negation tests whether an informal argument follows from a context, with emphasis on negations and deductive validity.The task uses argument schemes and asks whether the conclusion can be logically deduced.
- Natural-language understanding: Hyperbaton selects correct adjective order, while Salient Translation Error Detection classifies errors in English translations of German sentences.These tasks target grammatical ordering and cross-lingual translation assessment.
- Logical and mathematical reasoning: Multi-Step Arithmetic solves equations using basic arithmetic operations, while Object Counting asks for an item-class quantity.The arithmetic task is explicitly multi-step; the counting task aggregates possessions by class.
- Structured and spatial reasoning: Navigate, Tracking Shuffled Objects, and Logical Deduction require spatial reasoning over movement, swaps, or object placements.The tasks determine an agent’s return to its start, final object positions, or an ordering from spatial clues.
- Temporal reasoning: Temporal Sequences and Date Understanding infer times or dates from ordered events and date-related statements.The tasks ask when someone could perform an activity or compute a requested date.
B Per-Task Results
The paper presents scaling curves for answer-only and chain-of-thought prompting across BBH tasks, with model identities and prompting lengths specified for the evaluated systems.
- The InstructGPT models range from text-ada-001 through text-davinci-002.
- The Codex models include code-cushman-001 and code-davinci-002.
- The PaLM models evaluated are PaLM 8B, PaLM 62B, and PaLM 540B.
- OpenAI prompt target lengths are 1,000 for Geometric Shapes, 900 for four specified tasks, and 512 for all other tasks, while PaLM uses input length 2,048 and target length 512.
Multi-Step Arithmetic Two
The supplied material identifies Multi-Step Arithmetic Two within a collection of BBH task figures, alongside task labels and plotted model-series markers.
- Multi-Step Arithmetic Two is one of the tasks included in the BBH scaling figures.
- The surrounding figure material groups Multi-Step Arithmetic Two with Logical Deduction, Movie Recommendation, and Navigate.
- The supplied task labels also include Object Counting, Penguins in a Table, Reasoning about Colored Objects, and Ruin Names.
- Additional listed tasks include Salient Translation Error Detection, Snarks, Sports Understanding, Temporal Sequences, Tracking Shuffled Objects, Web of Lies, and Word Sorting.
C Full Set of CoT Prompts Used in Our Experiments
This appendix collects the chain-of-thought prompts used in the experiments, including a canary text and task-specific instructions and demonstrations across several reasoning domains.
- C Full Set of CoT Prompts Used in Our Experiments: The appendix states that it contains examples of BIG-Bench tasks likely to measure large language models.
- C Full Set of CoT Prompts Used in Our Experiments: A canary text is included to help prevent this document from entering future training data.
- C.3 CoT Prompt for Date Understanding: Date Understanding prompts ask models to infer dates from context and demonstrate step-by-step calendar reasoning.
- C.5 CoT Prompt for Dyck Languages: Dyck Languages prompts require completing parenthesis sequences while tracking stack configurations.
- C.7 CoT Prompt for Geometric Shapes: Geometric Shapes prompts ask models to name shapes from SVG paths by interpreting M and L commands and their coordinates.
- C.8 CoT Prompt for Hyperbaton: Hyperbaton prompts teach adjective ordering using categories such as opinion, size, age, shape, color, origin, material, and purpose.
- C.9 CoT Prompt for Logical Deduction (3/5/7): Logical Deduction prompts describe three objects in a fixed order and require deriving the requested position from consistent statements.
- C.10 CoT Prompt for Movie Recommendation: Movie Recommendation prompts ask for movies similar to a given list.
Salient Translation Error Detection
Salient Translation Error Detection asks models to identify the type of error introduced when translating a German source sentence into English.
- The task presents German-to-English translations containing a particular error and asks the model to classify that error.
- The listed error categories include named entities, numerical values, modifiers or adjectives, and negation or antonyms.
D Exact Task Filtering Criteria
BBH tasks were filtered using performance, task-format, data, and feasibility criteria, with many candidates discarded for violating those requirements or falling outside the study’s scope.
- Task-format criteria: Tasks were limited to those containing three or fewer subtasks.
- Discarded candidates: Additional candidates were discarded across numerous domains, including arithmetic, reasoning, language, social judgment, knowledge, and vision-related tasks.
- Data and evaluation criteria: Retained tasks had at least 100 input-output examples and reported human-rater or random performance, with human-rater performance strictly present.
- Task-format criteria: The task had to be classification-based or use exact match as its evaluation metric.
- Performance criteria: Tasks were retained only when the BIG-Bench paper’s best reported result was below average human-rater performance.
- Feasibility criteria: Tasks were excluded when they required specialized knowledge, exceeded a 60-minute solvability threshold, or were not worth attempting with chain-of-thought.