ARB: Advanced Reasoning Benchmark for Large Language Models

1 Introduction

• Rapid, sometimes unpredictable progress on reasoning tasks has made existing benchmarks less challenging and increased the need for harder evaluations.Source: §1 · p. 1
• Although models can pass entry-level exams in several fields, their understanding is reported to remain shallow and unreliable.The paper frames expert reasoning in specialized domains as important for automated systems intended to augment skilled professionals.Source: §1 · p. 2
• ARB tests expert reasoning in mathematics, physics, chemistry, biology, and law using graduate-level tasks from professional resources.The authors report that current models, including GPT-4, perform very poorly on its quantitative parts under standard prompting.Source: §1 · p. 2
• The benchmark includes hundreds of problems in quantitative subjects where LLMs are known to underperform.Source: §1 · p. 2
• Short-answer and open-response questions make up a large share of ARB, unlike earlier benchmarks dominated by multiple choice.Source: §1 · p. 2
• A rubric-based method lets GPT-4 score intermediate reasoning, with scores that track human grading surprisingly well but do not replace human evaluation.The authors also report that GPT-4-generated rubrics have good coverage.Source: §1 · p. 2

2 Related Work

• Prior reasoning work has often tested prompting and model enhancements on math or commonsense benchmarks, alongside newer planning and reasoning suites.Source: §2 · p. 2
• Existing benchmarks are described as limited in difficulty, reasoning coverage, and resemblance to complex real-world tasks.Recent advances such as Minerva are cited as evidence that some benchmarks may not offer sufficient challenge.Source: §2 · p. 2
• The paper examines rubric-based evaluation for complex math and physics problems amid growing use of LLMs as proxies for human evaluation.Rubrics had also been used for alignment evaluation.Source: §2 · p. 2

3 Benchmark

• Benchmark design prioritizes difficulty, useful skills, interpretable scores, reduced contamination, and a connection to general capabilities without pathological tasks.Source: §3 · p. 3

3.1 Formatting

• Questions are multiple choice, short answer, or open response, in descending order of prevalence; open responses require manual grading.Multiple-choice questions are easier to grade, while short answers use phrases or mathematical expressions and generally avoid algebraic expressions because equivalent forms complicate grading.Source: §3.1 · p. 3

3.2 Mathematics

• The mathematics subset spans contest and university-level problems, including topics such as representation theory and algebraic topology.It draws on problem books and exams, and excludes high-school contest problems already represented in other benchmarks.Source: §3.2 · p. 3

3.3 Physics

• Physics problems follow the mathematics format but include more numerical answers and sometimes figures, drawing on PhD qualifying-question materials.Source: §3.3 · p. 4

3.4 MCAT

• MCAT questions sampled from practice tests cover science and reading, with images retained for questions that use them.The source was chosen in part because few problems appear in standard web-searchable sources, limiting contamination.Source: §3.4 · p. 4

3.5 Law

• The law subset uses an older U.S. bar-exam practice set intended to test legal reasoning and broad legal knowledge while limiting likely training-data exposure.Source: §3.5 · p. 4

4 Evaluation

• The evaluation covers text-only problems using four models, task-specific instructions, and chain-of-thought prompting; answers are extracted after an ANSWER: delimiter and multiple-choice responses are matched to ground truth.Source: §4 · pp. 4–5
• Numerical answers are parsed with SymPy, symbolic expressions are checked for equivalence after normalization, and proofs are graded manually by authors with mathematics training.Symbolic equivalence checking is error-prone and limited to function-form responses; more advanced forms require human evaluation, as do natural-language proofs.Source: §4 · p. 5
• For symbolic and proof-like problems, GPT-4 creates a 10-point rubric and scores intermediate steps against a reference solution; rubric grading also provides denser evaluation and reduces human grading time.Source: §4 · p. 5

4.1 Results

• Models perform well on multiple-choice Law and MCAT subsets but struggle on numerical-answer questions; GPT-4 alone reliably simplifies complex expressions, yet falters on long-context arithmetic and symbolic manipulation.With partial credit for correctly deriving at least half of a problem’s subparts, GPT-4’s score on Proof-like problems rises to 16%.Source: §4.1 · pp. 5–6, Figure 1 · p. 6, §4.2 · p. 7
• GPT-3.5-turbo is the only model that cannot reliably follow multiple-choice answer-format instructions; its Law answers are unparsable about 25% of the time, versus under 5% for other models.GPT-4 is correctly parsed over 99% of the time.Source: §4.1 · p. 6
• Models also show a similarly low performance profile across symbolic problems, as reported in Table 2.Source: §4.1 · p. 6, Table 2 · p. 6

4.2 What Kind of Errors Do LLMs Make?

• GPT-4’s math and physics errors include misreading or partial answers, wrong approaches, logical errors, hallucinated facts or theorems, and arithmetic mistakes; logical errors and hallucinations are merged in Table 3.Source: §4.2 · pp. 6–7, Table 3 · p. 7
• Error attribution is approximate: larger subject subsets are sampled at 20–40 problems because categorization is fuzzy, and graders may underestimate arithmetic errors when the approach is clearly wrong or the source of an error is unclear.Source: §4.2 · p. 7

• GPT-4 generates rubrics from reference solutions, using few-shot human-written examples, then grades solutions against rubric-assigned intermediate steps.Source: §5 · p. 7
• GPT-4 rubrics covered crucial solution steps well but struggled to allocate points by importance, unlike GPT-3.5-turbo’s tendency to over-allocate to one or two steps.Source: §5 · p. 8
• GPT-4’s rubric-based self-evaluation had moderately high correlation with manual scores and almost never differed from ground truth by more than two points.Source: Table 5 · p. 8, §5 · p. 8
• The main failure mode was partial credit for attempted solutions outside the problem rubric, which human evaluators scored zero.Source: §5 · p. 8
• GPT-4 performed best on physics intermediate reasoning, recalling useful concepts but struggling with mathematical manipulation; it was weaker at planning in math, especially proofs.Source: §5 · p. 8

• ARB evaluates advanced reasoning with problems from the sciences and law, sourced from graduate-level exams and professional resources.Source: §6 · pp. 8–9
• Despite recent advances, current LLMs perform very poorly on ARB’s quantitative subjects, and rubric-based self-evaluation is not yet reliable enough to replace human grading.Source: §6 · p. 9
• Data contamination is possible because models may have been trained on material included in benchmarks, including MCAT books.Source: §6 · p. 9
• The authors hope ARB’s higher difficulty standards will help the research community ground assessments of increasingly powerful models more accurately.Source: §6 · p. 9
• ARB does not cover all human abilities; even perfect benchmark performance could coexist with much weaker ability than most educated people in many areas.Source: §6 · p. 9

A Datasheet

• The authors present the data card using the format proposed by Pushkarna et al. [2022].Source: A Datasheet · p. 13
• The dataset is described as having little associated risk because it targets benchmarking and is too small to advance model capabilities.Source: A Datasheet · p. 13
• Maintenance is limited to technical issues and bug or error fixes; no new versions are planned.Source: A Datasheet · p. 13
• The dataset includes no sensitive human attributes, and its text entries are stored in LaTeX.Source: A Datasheet · p. 14

B Dataset format

• Each dataset entry pairs a problem statement with a ground-truth solution, and the JSONL files also contain final answers and available metadata.Metadata includes subject names and problem topics where available.Source: B Dataset format · p. 14, A Datasheet · p. 14
• ARB focuses primarily on math and physics, with law and MCAT included to assess models across a wider range of technical domains.The authors describe math and physics as more challenging than existing benchmarks.Source: B Dataset format · p. 14

• Problems can be accessed through API calls by split and subject area.Source: C Accessing the dataset · p. 14
• The authors host the data through an API rather than popular public platforms such as HuggingFace or GitHub because of contamination concerns.They note that training on benchmark datasets could inadvertently inflate performance on those benchmarks.Source: C Accessing the dataset · p. 14
• The authors considered restricting access to web crawlers the only way to ensure the dataset’s integrity for future use.Source: C Accessing the dataset · p. 14

D Model-based Rubric Eval

• GPT-4’s rubric covers important reference-solution subparts and distributes points evenly, whereas GPT-3.5-turbo often underspecifies subproblems and rewards irrelevant steps.Source: D Model-based Rubric Eval · p. 14
• Rubric grading can credit useful intermediate ideas even when an earlier error leads GPT-4 to an incorrect final answer.In the example, GPT-4 identifies an ellipse parameterization formula and that an isosceles triangle optimizes the area, but fails to compute the answer correctly.Source: D Model-based Rubric Eval · p. 14
• The section directs readers to API access instructions and documentation.Source: D Model-based Rubric Eval · p. 14

D.1 Using ChatGPT for Symbolic Evaluation

• Final-answer grading is less costly than grading intermediate steps, but more involved symbolic expressions still require human evaluation for accurate comparison.Source: §D.1 · p. 15
• GPT-3.5-turbo compares extracted GPT-4 symbolic answers with reference answers, providing an automated approach to symbolic equivalence grading.Source: §D.1 · p. 15
• A false-positive rate of 0 comes with missed correct credits, particularly for expressions with more than three variables.Source: §D.1 · p. 15
• GPT-3.5-turbo may fail to match a correct answer when GPT-4 presents it in a format different from the reference.Providing the full reasoning trace might avoid some errors, but the authors expect little benefit and possible harm.Source: §D.1 · p. 15

E Interesting Examples

• In a math example, GPT-4 performs formal manipulations that do not simplify the problem, then relies on a black-box numerical calculation for its incorrect answer.Source: E Interesting Examples · p. 15
• In a DNA-replication question, GPT-4 confuses DNA molecules with strands and incorrectly concludes that radioactive thymine must occur in every strand.It consequently selects D rather than recognizing that radioactivity need only be present somewhere in each resulting double helix.Source: E Interesting Examples · p. 15
• In the law example, GPT-4 treats a claim stated in an answer choice as a fact and selects the choice affirming that identical crossing offers form an offer and acceptance.Source: E Interesting Examples · p. 15

F Memorization

• The authors say leakage and memorization are hard to quantify; examples suggest they may matter, but low overall performance limits their impact.GPT-3.5-turbo appears to recall a capacitance formula despite faulty reasoning, while GPT-4 follows its reasoning chain and gives a slightly inaccurate answer.Source: F Memorization · p. 16

G Prompts for Sections 4 and 5

• OpenAI chat models receive a prepended system prompt, while other models do not because there is no established approach and it might hurt performance.The authors expect little effect from omitting it because the user prompts restate its directions.Source: G Prompts for Sections 4 and 5 · p. 16

H Checklist Information.

• ARB is intended primarily for research, not to train models that help students cheat, and the dataset and helper code use CC BY 4.0 and MIT licenses, respectively.The authors cite Fair Use §107 for problems originating in books listed in Section 3.Source: H Checklist Information · p. 16
• In the example model-generated rubric evaluation in Table 7, the model makes good partial progress but remains unable to arrive at the correct final answer.Source: Table 7 · p. 17
• The ellipse-triangle evaluation awards 4 points for correct parameterization and identifying the equilateral maximizer, despite an incorrect final area.The rubric specifies the determinant formula and connecting the problem to the unit circle as additional criteria.Source: H Checklist Information · pp. 17, 19
• GPT-3.5-turbo’s symbolic-equivalence evaluation had a false positive rate of 0 but allowed many false negatives.The caption says the model often failed to recognize equivalent expressions.Source: Table 8 · p. 19
• In benchmark examples, GPT-4 often returned numerical answers after internal black-box calculations that were rarely correct, and on an MCAT question it mistook two DNA copies for two amino-acid strands.The cited set also includes a law question and an ARB memorization example: GPT-3.5-turbo’s response indicates memorization of the correct expression, while GPT-4 reasons to a wrong final answer.Source: Table 9 · p. 20, Table 10 · p. 21, Table 11 · p. 22
• The prompt examples cover multiple-choice MCAT and law questions, numerical problems, symbolic problems, and proof-like problems.Multiple-choice questions have four to five options and are sourced from standardized tests including the MCAT and bar-exam preparation.Source: Table 13 · p. 25, Table 14 · p. 25, Table 15 · p. 26
• Additional prompt examples specify GPT-3.5-turbo symbolic evaluation, GPT-4 rubric self-evaluation, and GPT-4 rubric design.Source: Table 17 · p. 27, Table 18 · p. 27, Table 19 · p. 28