Source-linked AI summary
BIG-Bench Extra Hard
Mehran Kazemi, Bahare Fatemi, Hritik Bansal, John Palowitch, Chrysovalantis Anastasiou, Sanket Vaibhav Mehta, Lalit K. Jain, Virginia Aglietti, Disha Jindal, Peter Chen, Nishanth Dikkala, Gladys Tyen, Xin Liu, Uri Shalit, Silvia Chiappa, Kate Olszewska, Yi Tay, Vinh Q. Tran, Quoc V. Le, Orhan Firat
TL;DR
BBH has become saturated, limiting its ability to distinguish the reasoning abilities of frontier LLMs. BBEH replaces its 23 tasks with substantially harder counterparts and finds that advanced models still perform poorly, revealing substantial room for improvement.
Problem
BBH has become saturated, with state-of-the-art models exceeding 90% accuracy and becoming harder to distinguish by reasoning ability.
Method
BBEH replaces each of BBH’s 23 tasks with a novel, significantly harder counterpart targeting similar reasoning capabilities.
Results
23.9% ceiling accuracy for the best general-purpose model and 54.2% for the reasoning-specialized model show that BBEH remains challenging.
Takeaways & Limitations
BBEH offers a more rigorous platform for studying robust and versatile reasoning in LLMs and guiding future research.
Takeaways & Limitations
Adversarial construction against two reference models may bias BBEH toward particular failure modes and complicate fair comparisons with those models.
Abstract
from arXiv · showhide
Large language models (LLMs) are increasingly deployed in everyday applications, demanding robust general reasoning capabilities and diverse reasoning skillset. However, current LLM reasoning benchmarks predominantly focus on mathematical and coding abilities, leaving a gap in evaluating broader reasoning proficiencies. One particular exception is the BIG-Bench dataset, which has served as a crucial benchmark for evaluating the general reasoning capabilities of LLMs, thanks to its diverse set of challenging tasks that allowed for a comprehensive assessment of general reasoning across various skills within a unified framework. However, recent advances in LLMs have led to saturation on BIG-Bench, and its harder version BIG-Bench Hard (BBH). State-of-the-art models achieve near-perfect scores on many tasks in BBH, thus diminishing its utility. To address this limitation, we introduce BIG-Bench Extra Hard (BBEH), a new benchmark designed to push the boundaries of LLM reasoning evaluation. BBEH replaces each task in BBH with a novel task that probes a similar reasoning capability but exhibits significantly increased difficulty. We evaluate various models on BBEH and observe a (harmonic) average accuracy of 9.8\% for the best general-purpose model and 44.8\% for the best reasoning-specialized model, indicating substantial room for improvement and highlighting the ongoing challenge of achieving robust general reasoning in LLMs. We release BBEH publicly at: https://github.com/google-deepmind/bbeh.
1. Introduction
BBEH responds to saturation in BBH by replacing its tasks with harder counterparts that test similar capabilities. Evaluation shows substantial remaining difficulty for both general-purpose and reasoning-specialized models.
- BBH has become saturated, with state-of-the-art models exceeding 90% accuracy and reduced discrimination among current LLMs.
- BBEH replaces all 23 BBH tasks with novel, significantly harder tasks targeting similar reasoning capabilities.The benchmark targets skills including many-hop reasoning, long-context processing, distractor handling, and pattern induction.
- 23.9% ceiling accuracy was observed for the best general-purpose model and 54.2% for the best reasoning-specialized model.The evaluation also reports failure analysis for both model types.
2. Related Work
Prior evaluation has emphasized mathematical and scientific reasoning, while the generalizability of those skills to broader reasoning domains remains unclear. The paper positions open, diverse evaluation as a continuing need.
- Mathematical and STEM-focused benchmarks dominate LLM reasoning evaluation, while transfer to broader domains remains unclear.
- BBEH contributes an open evaluation framework with automatic and deterministic correctness checking across multiple fine-grained tasks.
3. Going Beyond BIG-Bench Hard
BBEH uses BBH as a guide while addressing limitations involving chance performance, shortcuts, short inputs, shallow reasoning, and limited skill coverage. Its design expands the range and depth of tested reasoning abilities.
- BBH is valuable for diverse, automatically verifiable reasoning tasks but has become less suitable as newer LLMs improve.
- BBH has high chance baselines, exploitable shortcuts, roughly 700-character macro-average inputs, few reasoning hops, and limited skill coverage.
- BBEH aims to lower chance performance, remove shortcuts, lengthen inputs, require more reasoning hops, and preserve automatic verification.
- BBEH expands coverage to many-hop reasoning, very long-range dependencies, long-context processing, needle-in-a-haystack retrieval, distractor resistance, and knowledge-intensive reasoning.
4. BIG-Bench Extra Hard
BBEH is constructed as a harder, diverse successor to BBH through task replacement and semi-adversarial difficulty design. Its construction uses reference-model feedback while acknowledging resulting benchmark biases.
- BBEH replaces each BBH task with a more challenging task in a similar reasoning domain, preserving the original benchmark’s diversity.
- Task difficulty was iteratively increased using Gemini 1.5 Flash and Gemini Thinking Experimental as general-purpose and reasoning-specialized reference models.
- The construction process sometimes inspected model strategies to block shortcuts, such as replacing Boolean literals with semantically equivalent factual sentences.
- The benchmark includes 23 tasks described by their tested abilities, reasoning capabilities, construction methods, and BBH counterparts.
- Reference-model choice biases BBEH toward certain failure modes, and adversarial construction can hinder fair comparisons between reference and non-reference models.
5. Results and Analyses
BBEH analysis compares its task inputs, model performance, and reasoning-related gains across model families. The benchmark uses longer contexts and outputs than BBH, while substantial performance gaps remain, especially for general-purpose models.
- Input/Context Length: BBEH’s macro average context length is about six times larger than BBH’s across almost all tasks.
- Required Amount of Thinking: BBEH’s macro average output length is about seven times larger than BBH’s for Gemini 2.0 Flash responses.Output length is used as a proxy for the amount of thinking required.
- Models: 9.8% harmonic mean accuracy is the best result for general-purpose models, compared with 44.8% for reasoning-specialized models on BBEH.The reference reasoning model reaches 20.2% harmonic mean accuracy, while the random baseline is 8.4%.
- Models: 23.9% micro average accuracy is achieved by the best general-purpose model, compared with 54.2% for the best reasoning-specialized model.Micro averages are reported separately from the harmonic mean results.
- General-Purpose vs Reasoning Models: Reasoning models gain most on counting, planning, arithmetic, and algorithms, but least on commonsense, humour, sarcasm, and causation.These results indicate larger gains on formal problems than on softer reasoning skills.
- The Effect of Model Size: Larger models show the greatest gains on many-hop reasoning and algorithms, while humour, commonsense, and causal reasoning show the least gains.SARC Triples is an exception, with large gains potentially reflecting its composition of three sub-questions.
6. Conclusion
BBEH is introduced as a more challenging successor to BBH because existing general-reasoning benchmarks have approached ceiling performance. Its results expose continued difficulty in diverse reasoning and provide a platform for future evaluation and improvement.
- BBEH addresses near-ceiling performance on BIG-Bench and BBH by amplifying task difficulty while preserving their core diversity.
- The benchmark shows that even advanced LLMs still struggle with fundamental aspects of general reasoning.
- The context-length and output-length analysis excludes Shuffled Objects because some models exhausted effective tokens and degenerated.
- BBEH offers a more rigorous platform for research aimed at improving LLMs in complex, real-world applications.
A. Detailed Description of the Tasks and Task-Specific Insights from Experiments
This section describes how the 23 new BBEH tasks were created and reports task-specific experimental insights.
- The paper details the creation of all 23 BBEH tasks and provides task-specific insights from experiments.
A.1. BoardgameQA
BoardgameQA tests multi-hop reasoning and conflict resolution over defeasible theories, with models selecting proved, disproved, or unknown. Its difficulty is controlled by reasoning depth, and models often over-predict unknown when proofs exist.
- BoardgameQA supplies facts, potentially contradictory rules, and rule preferences for determining whether a queried statement is proved, disproved, or unknown.When conflicting rules apply, the higher-preference rule determines the conclusion.
- 33.3% is the random baseline because each question has three possible labels.
- Depth measures the number of reasoning hops needed to compute the statement’s truth value; the benchmark samples tasks at depths 6, 7, and 8.Prompts were clarified for zero-shot use and uniformly sampled across depths and labels.
- Models over-predict unknown even though only one-third of problems have that label.This reflects difficulty searching facts and rules to find an available proof or disproof.
A.4. Causal Understanding
Causal Understanding evaluates whether models align with human judgments of causal relations, including ambiguity and necessary or sufficient causes. Gemini 2.0 Flash performs better on cause questions than causal judgments but struggles especially with ambiguous or absent causal relations.
- The task uses causal stories labeled Yes, No, or Ambiguous according to human judgments, with Ambiguous assigned when annotators strongly disagree.Models are instructed to predict how the majority of people would answer.
- The benchmark adds 58 necessary-and-sufficient-cause examples, asking whether an actor is necessary or sufficient for an event.These examples were modified to resolve ambiguities and incorrect outputs, with three experts agreeing on minimal changes.
- 45% of Gemini 2.0 Flash’s causal-judgment queries were correct, versus 71% on necessary-and-sufficient-cause queries.Random performance for the causal-judgment queries is 33%.
- 44 of 45 Ambiguous errors and 24 of 49 No errors show difficulty recognizing absent or uncertain causal relationships.Only 10 of 48 Yes examples were errors.
- The model made 11 errors on sufficient-cause examples and 6 on necessary-cause examples despite usually recalling the definitions and using counterfactual reasoning.
A.6. Dyck Languages
The Dyck Languages adaptation asks models to locate the first error in an existing chain-of-thought trace or identify that no error exists. Frontier models usually miss the earliest error rather than falsely flagging an earlier correct step.
- The task identifies the first mistake in a chain-of-thought sequence generated for a Dyck Languages problem, or reports that no mistake exists.The target answer is the number of the first erroneous step.
- Errors arise either from misclassifying a correct step before the first error or from missing the first error and selecting a later one.
- 98.7% of o3-mini (high) errors belonged to the later-error category, compared with 100% for Gemini 2.0 Flash, 94.9% for Gemini 2.0 Flash-Lite, and 96.8% for GPT4o.The reported pattern indicates that models can identify correct reasoning steps but fail to identify erroneous ones.
- The geometric-shape task increases difficulty by varying line breaks and whether added lines intersect to form shapes.It creates four subsets from the cross product of few versus many breaks and intersect versus no intersect.
- o3-mini (high) scored 58% with few breaks versus 47% with many, and 72% without intersections versus 33% with intersections.Both modifications are reported as increasing task difficulty.
A.8. Hyperbaton
The Hyperbaton replacement tests whether models can infer adjective order in a newly defined English variant rather than rely on standard English knowledge. Models may recover partial orderings yet apply prior ordering assumptions when selecting answers.
- BBH’s Hyperbaton task tests knowledge of English adjective categories and their prescribed ordering before nouns.The standard order is opinion → size → age → shape → color → origin → material → purpose.
- BBEH replaces Hyperbaton with a task combining linguistic knowledge and inductive reasoning over a new adjective-order variant.
- The variant randomly shuffles adjective order, provides 50–250 examples, and asks models to select all correct sentences from 10 options.Each sentence contains 1 to 3 adjectives, and “None of the above” is a distractor.
- Some models infer partial adjective orderings but rely on standard English ordering when selecting answers.
- The model correctly identified option D but also selected A, C, E, H, and I after incorrectly assuming material follows origin.The model’s partial ordering was otherwise consistent with the examples.
- Linguini problems require meta-linguistic awareness and deductive reasoning, and BBEH samples translation, fill-blanks, number-to-text, and text-to-number categories.The benchmark converts multi-answer items into single-answer questions by selecting one blank.
A.12. NYCC
The NYCC task increases difficulty by replacing pairwise caption comparisons with a multiple-choice selection among captions spanning extreme rating ranks.
- Task construction: The NYCC dataset contains cartoons, humorous captions, and crowdsourced ratings ranging from “Unfunny” to “Funny”.
- Task construction: Earlier pairwise caption-comparison tasks presented two captions and achieved approximately 70% accuracy for GPT-4 Turbo.
- Task construction: For each contest, BBEH samples one top-ten caption and compares it against captions ranked 1000–1009, asking the model to choose the funniest.
- Difficulty: Adding multiple possible captions makes the NYCC task significantly more challenging than the pairwise formulation.
A.22. Word Sorting
Word Sorting combines chain-of-thought error detection with sorting under a modified alphabet, exposing failures when models must override familiar ordering priors.
- Task design: The Word Sorting task is divided into two sub-tasks.
- Mistake detection: The first sub-task asks models to identify the first mistake in a chain-of-thought sequence or determine that no mistake exists.
- Modified alphabet: The second sub-task sorts words using a newly specified alphabet order, including swapped, moved, or completely reordered letters.
- Failure mode: A model correctly reproduced the modified alphabet but sorted the words according to the original order instead.
C. Reproducibility
The evaluation pipeline uses documented API or local-GPU execution and standardized answer extraction to support reproducible model comparisons.
- Execution: Most model results were obtained through specified APIs, while publicly available models were loaded directly onto GPUs.
- Answer format: A fixed instruction asks models to reason step by step and prefix final answers with “The answer is:”.
- Extraction: The evaluator searches several accepted answer prefixes and applies minimal normalization before extracting responses.
- Extraction: Comma-separated labels are accepted when answer spacing differs only by replacing spaces after commas with blanks.