PaperlayerView exact revision on arXiv

Source-linked AI summary

OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI

Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, Yikai Zhang, Yuqing Yang, Ting Wu, Binjie Wang, Shichao Sun, Yang Xiao, Yiyuan Li, Fan Zhou, Steffi Chern, Yiwei Qin, Yan Ma, Jiadi Su, Yixiu Liu, Yuxiang Zheng, Shaoting Zhang, Dahua Lin, Yu Qiao, Pengfei Liu

arXiv:2406.12753v2cs.CLcs.AI

TL;DR

Existing benchmarks leave limited coverage of Olympic-level multidisciplinary cognitive reasoning and process-level evaluation. OlympicArena brings together 11,163 bilingual problems across seven fields and 62 competitions, with answer- and process-level analyses; GPT-4o achieves 39.97% overall accuracy.

  • Problem

    Existing benchmarks largely emphasize knowledge-intensive or simple concept-application tasks and answer-level evaluation, leaving Olympic-level multidisciplinary cognitive reasoning insufficiently assessed.

    Source: §1 · p. 2, §2 · p. 4

  • Method

    OlympicArena assembles bilingual, text-only and interleaved text-image Olympic problems across seven disciplines and evaluates answers, reasoning steps, and logical and visual abilities.

    Source: §1 · p. 3, §3.1 · p. 5

  • Results

    Current models struggle on OlympicArena’s complex multidisciplinary tasks; GPT-4o achieves only 39.97% overall accuracy.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors present OlympicArena as a benchmark and stepping stone for future AI advances in science and engineering.

    Source: §5 · p. 10

    Yi-VL-34B is excluded from one comparison because it does not support multiple image inputs.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

The evolution of Artificial Intelligence (AI) has been significantly accelerated by advancements in Large Language Models (LLMs) and Large Multimodal Models (LMMs), gradually showcasing potential cognitive reasoning abilities in problem-solving and scientific discovery (i.e., AI4Science) once exclusive to human intellect. To comprehensively evaluate current models' performance in cognitive reasoning abilities, we introduce OlympicArena, which includes 11,163 bilingual problems across both text-only and interleaved text-image modalities. These challenges encompass a wide range of disciplines spanning seven fields and 62 international Olympic competitions, rigorously examined for data leakage. We argue that the challenges in Olympic competition problems are ideal for evaluating AI's cognitive reasoning due to their complexity and interdisciplinary nature, which are essential for tackling complex scientific challenges and facilitating discoveries. Beyond evaluating performance across various disciplines using answer-only criteria, we conduct detailed experiments and analyses from multiple perspectives. We delve into the models' cognitive reasoning abilities, their performance across different modalities, and their outcomes in process-level evaluations, which are vital for tasks requiring complex reasoning with lengthy solutions. Our extensive evaluations reveal that even advanced models like GPT-4o only achieve a 39.97% overall accuracy, illustrating current AI limitations in complex reasoning and multimodal integration. Through the OlympicArena, we aim to advance AI towards superintelligence, equipping it to address more complex challenges in science and beyond. We also provide a comprehensive set of resources to support AI research, including a benchmark dataset, an open-source annotation platform, a detailed evaluation tool, and a leaderboard with automatic submission features.

1 Introduction

OlympicArena addresses gaps in existing cognitive-reasoning benchmarks with a broad, bilingual, multimodal Olympic-level dataset and evaluations of both answers and reasoning processes. Results show that even leading models struggle with its complex tasks.

1 Introduction

  • Existing benchmarks largely emphasize knowledge-intensive or simple concept-application problems, text-only assessment, and answer correctness, motivating broader and more fine-grained evaluation.A cited Olympic-level benchmark covers only mathematics and physics.Source: §1 · p. 2
  • OlympicArena contains 11,163 bilingual problems across seven disciplines and 62 competitions, in text-only and interleaved text-image modalities.It adds process-level evaluation of step-by-step reasoning alongside answer-level assessment.Source: §1 · p. 3
  • Evaluations combine rule- and model-based answer scoring with step-level process scoring and analyses of logical and visual reasoning abilities.The experiments cover proprietary and open-source LMMs and LLMs in text-only and image-caption settings.Source: §1 · p. 3
  • 39.97% overall accuracy is GPT-4o’s result, while open-source models struggle to reach 20%.The authors also report that current LMMs show no notable gains from interleaved visual information over text-only counterparts.Source: §1 · p. 3
  • Fine-grained analyses find weaknesses in decompositional reasoning, spatial and geometric perception, abstract-symbol understanding, and use of interleaved visual information.Process evaluation also finds that most models correctly execute some reasoning steps despite giving incorrect answers; detected leakage is rare and does not consistently yield correct solutions.Source: §1 · p. 3

2 Related Work

Related work traces benchmarks from specialized evaluations to multidisciplinary and Olympic-level challenges, while identifying gaps in reasoning difficulty and process-level assessment. OlympicArena is positioned as a broader benchmark with more fine-grained evaluation.

Benchmark AI Intelligence

  • Existing multidisciplinary benchmarks such as MMLU, AGIEval, C-Eval, and CMMLU emphasize knowledge-intensive tasks, which the authors say have become insufficient as LLMs progress.The related-work discussion also notes that some benchmarks target university-level or multimodal problems but remain focused on knowledge or simple concept application.Source: §2 · pp. 3–4

Benchmark comparison

  • Table 1 compares benchmarks by subjects, modality, language, size, answer types, evaluation methods, leakage detection, and difficulty proportions.The accompanying difficulty labels are knowledge recall, concept application, and cognitive reasoning.Source: Table 1 · p. 4, §2 · p. 4

Benchmark AI Intelligence

  • Olympic-level and algorithmic challenges have entered benchmark research, but the authors identify no Olympic-level multidisciplinary benchmark that comprehensively tests cognitive problem-solving.A concurrent Olympic-level benchmark is limited to mathematics and physics.Source: §2 · p. 4

Rigorous Evaluation for Reasoning

  • Most existing benchmarks emphasize answer-level evaluation, while few incorporate process-level assessment of reasoning reliability and faithfulness in lengthy solutions.OlympicArena adds process-level evaluation to help developers understand model reasoning behavior.Source: §2 · p. 4

3 The OlympicArena Benchmark

OlympicArena combines a broad Olympic-level problem collection with multimodal and bilingual coverage, detailed reasoning annotations, and a defined train-and-test resource split. Its construction includes collection, annotation, validation, and deduplication steps.

3.1 Overview

  • The benchmark spans seven disciplines, 34 specialized branches, and bilingual text-only and interleaved text-image modalities.Its scope is designed to assess LLMs’ and LMMs’ cognitive reasoning on Olympic-level problems.Source: §3.1 · p. 4
  • 11,163 problems from 62 Olympic competitions cover 13 answer types, including objective and subjective formats such as programming tasks.The authors also conduct data-leakage detection experiments on several models.Source: §3.1 · p. 5

3.2 Data Collection

  • Competition problems are collected from publicly available PDFs, converted to Markdown with Mathpix, and supplemented with test cases for computer-science programming problems.The authors state that collection follows copyright and licensing considerations.Source: §3.2 · p. 5

3.3 Data Annotation

  • Annotators extract problems and available solutions, then apply multi-step validation and within-competition deduplication; GPT-4V labels difficulty and logical and visual reasoning abilities.Difficulty categories are knowledge recall, concept application, and cognitive reasoning; the ability taxonomy includes eight logical and five visual types.Source: §3.3 · p. 5

3.4 Data Splitting

  • OlympicArena-ot contains 548 problems, OlympicArena-val samples 638 problems, and the remaining problems form the official test set.The validation set includes step-by-step solutions, and reported results use the entire benchmark dataset.Source: §3.4 · p. 6

4 Experiments

Experiments evaluate models on OlympicArena using answer-level and process-level measures across text-only, image-caption, and multimodal settings. GPT-4o reaches 39.97% overall accuracy, while fine-grained analyses reveal reasoning and visual-input weaknesses, and leakage checks indicate minimal leakage.

4.1 Experimental Setup

  • Experiments compare interleaved multimodal inputs, image captions replacing images, and text-only baselines under zero-shot prompts.For models unable to handle interleaved inputs, multiple images are concatenated into one input.Source: §4.1 · p. 6

4.2 Evaluation

  • Answer scoring combines rule-based matching, pass@k for code, and GPT-4V review for other answers; process scoring rates structured steps, with 83% human inter-annotator agreement.Process-level evaluation samples 96 problems with reference solutions and scores each structured reasoning step from 0 to 1.Source: §4.2 · p. 6

4.3 Main Results

  • 39.97% overall accuracy for GPT-4o, while open-source models struggle to reach 20%; mathematics, physics, and programming competitions remain especially difficult.Table 3 reports experimental results as percentages; for code generation, a problem counts as correct if any generated code passes all test cases.Source: §4.3 · pp. 6–7, Table 3 · p. 7

4.4 Fine-grained Analysis

  • Models tend to perform better on abductive and cause-and-effect reasoning and pattern recognition and comparative visualization, but worse on inductive and decompositional reasoning and spatial, geometric, or abstract-symbol tasks.The figure categorizes eight logical and five visual reasoning abilities.Source: §4.4 · p. 7, Figure 3 · p. 8
  • Few LMMs, including GPT-4o and Qwen-VL-Chat, improve significantly with images; others may prioritize text, lose language ability, or struggle with interleaved formats.Some LMMs show decreased effectiveness with image inputs; these explanations are presented as possible reasons.Source: §4.4 · pp. 7–8, Figure 4 · p. 8
  • Answer-level and process-level scores generally align, but process accuracy is often higher; errors occur more often late in reasoning, and some disciplines show stronger answer than process performance.The authors suggest models may overlook the reasonableness of intermediate steps even when those steps do not affect the final result.Source: §4.4 · pp. 8–9, Figure 4 · p. 8
  • Yi-VL-34B is excluded from comparisons because it does not support multiple image inputs, which could make the comparison unfair.Source: §4.4 · p. 8

4.5 Efforts on Data Leakage Detection

  • Some base or text-only models may have encountered benchmark instances, including 43 for Qwen1.5-32B-Chat, yet corresponding chat models answer even fewer; the authors conclude leakage is minimal.The figure reports detected leaked samples and correct responses from corresponding text-only and multimodal chat models.Source: §4.5 · p. 9, Figure 6 · p. 9

5 Conclusion

OlympicArena evaluates LLM and LMM cognitive reasoning on Olympic-level problems, where even GPT-4o struggles with complex problem solving. The authors hope it will support future AI advances in science and engineering.

5 Conclusion

  • OlympicArena is a benchmark for evaluating LLM and LMM cognitive reasoning on Olympic-level problems.Source: §5 · p. 10
  • GPT-4o does not perform well when applying cognitive reasoning to solve complex problems.Source: §5 · p. 10
  • The authors hope OlympicArena will serve as a stepping stone for future AI advances in science and engineering.Source: §5 · p. 10

A Detailed Statistics of the Benchmark

OlympicArena documents its competition sources, subject coverage, languages, modalities, answer types, and image types through accompanying tables and figures. Its answer taxonomy contains 13 types, and its multimodal scientific problems are organized into five image types.

A.1 Distribution of Problems

  • OlympicArena reports benchmark distributions across disciplines and modalities, with competition sources and subject subfields detailed separately.Table 4 identifies competitions partially sourced from OlympiadBench or MMcode; Table 6 presents statistics across disciplines and modalities.Source: §A.1 · p. 15, Table 4 · p. 16, Table 5 · p. 17

A.2 Answer Types

  • The benchmark uses 13 answer types, distilled from observations of problems and prior benchmarks to cover as many problems as possible.Their definitions are provided in Table 7.Source: §A.2 · p. 15, Table 7 · p. 18

A.3 Image Types

  • Multimodal scientific problems are grouped into five image types, with examples and their distribution presented in the appendix.Figure 7 gives examples, Figure 8 shows the distribution, and Table 8 provides definitions and examples.Source: §A.3 · p. 15, Figure 7 · p. 15, Figure 8 · p. 17

B Data Annotation

The appendix describes how OlympicArena problems are annotated, categorized by difficulty and reasoning abilities, and presented to evaluated models. It also specifies the evaluated model sets, prompt design, and generation settings.

B.1 Problem Extraction and Annotation

  • OlympicArena annotations use a Streamlit interface, rule-based checks, and a multi-step review to identify and correct errors.Approximately 30 university students annotate problems; a preliminary rule-based check is followed by review from different annotators.Source: §B.1 · p. 19

B.2 Annotation for Difficulty Levels

  • Problems are categorized as Knowledge Recall, Concept Application, or Cognitive Reasoning, according to the level of recall, application, or reasoning required.The categories distinguish factual recall, straightforward concept use, and logical or visual reasoning.Source: §B.2 · p. 20

B.3 Cognitive Reasoning Abilities Annotation

  • The annotation scheme distinguishes eight logical reasoning abilities and five visual reasoning abilities, including pattern recognition, spatial reasoning, and symbol interpretation.The logical categories include deduction, induction, abduction, analogy, cause-and-effect, critical thinking, decomposition, and quantitative reasoning; the visual categories include pattern, spatial, diagrammatic, symbol, and comparative visualization abilities.Source: §B.3 · pp. 20–21

C.1 Prompt for Image Caption

C.2 Models

  • Experiments include proprietary and open-source LMMs and corresponding text models, using APIs for proprietary models and an 8-card A800 cluster for open-source models.Gemini-1.5-pro was excluded because API access had significant rate limits during the experiments.Source: §C.2 · pp. 23–24, Table 9 · p. 24

C.3 Evaluation Prompts

  • Model prompts are tailored to answer types, with output formats specified for each type and optional context for problems relying on earlier questions.The prompt template and answer-type instructions are provided in Figure 15 and Table 10.Source: §C.3 · p. 24, Figure 15 · p. 24

C.4 Model Hyperparameters

  • Generation uses a 2048-token output limit and temperature 0.0, except code-generation tasks use temperature 0.2.Source: §C.4 · p. 25

C.5 Answer-level Evaluation Protocols

Answer-level evaluation combines rule-based matching with GPT-4V judging for answers that cannot be handled appropriately by rules, while process-level evaluation scores individual reasoning steps. The appendix also points to result tables organized by reasoning abilities, multimodal subjects, and process-level performance.

C.5 Answer-level Evaluation Protocols

  • Fixed-answer problems are evaluated by extracting boxed answers and matching them according to answer type; other answer types are assessed by GPT-4V using the problem, reference answer, optional solution, and model response.A sample of 100 model-evaluated problems had nearly 80% agreement with human evaluations; because these problems account for about 5% of the total, the authors estimate an overall error rate of around 1%.Source: §C.5 · p. 25
  • Answer-type rules include scientific-notation tolerance for estimated numeric answers, symbolic comparison for expressions and equations, and exact interval-boundary checks.For matching exponents, estimated numeric answers may differ by 0.1 in the coefficient; SymPy is used for expressions and equations.Source: §C.5 · p. 25
  • Multiple-quantity answers follow the prompted output order, multiple-answer problems require all possible answers, sets and tuples are checked positionally or identically, and code is tested against provided cases.For code tasks, the protocol uses unbiased pass@k with k = 1 and n = 5, where c is the number of samples passing all test cases.Source: §C.5 · p. 25

C.6 Process-level Evaluation Protocols

  • Process-level evaluation reformats gold and generated solutions step by step, scores each model-output step as correct or incorrect, and averages the step scores per problem.Source: §C.6 · p. 26

D.1 Results across Logical and Visual Reasoning Abilities

  • Table 11 and Table 12 report model performance across logical and visual reasoning abilities.Source: §D.1 · p. 26

D.2 Results on Multimodal Problems

  • Table 13 reports model performance on multimodal problems across subjects.Source: §D.2 · p. 26

D.3 Process-level Evaluation Results

  • Table 14 reports process-level results across subjects.Source: §D.3 · p. 26

D.4 Results across Different Languages

Table 15 reports experimental results across English and Chinese in percentage form.

D.4 Results across Different Languages

  • Table 15 reports experimental results across English and Chinese in percentages.The supplied passage does not provide the table’s values or a comparison between languages.Source: §D.4 · p. 26

E Data Leakage Detection Details

The authors detect possible training-data leakage using n-gram prediction and report minor leakage across subjects. Tables 16–18 distinguish detected leaks from correct answers by text-only and multimodal chat models.

E Data Leakage Detection Details

  • Leakage detection combines exact match, edit distance, and ROUGE-L checks on predicted next 5-grams from sampled starting points.A prediction counts as correct if edit distance or ROUGE-L similarity exceeds 75%; instances detected by any metric are unioned.Source: E Data Leakage Detection Details · p. 27
  • Detected leakage was minor across subjects, and some cases found for base models were no longer detectable in their chat versions.The authors hypothesize that dialogue optimization may affect next-token prediction ability and performance.Source: E Data Leakage Detection Details · p. 27
  • Tables 16–18 present full data-leakage detection results for base or text-only chat models behind evaluated models, including leakage-instance counts and counts correctly answered by corresponding text-only and multimodal chat models.“# Leak.” denotes the number of leakage instances; “# T” and “# MM” represent instances correctly answered among these leaks by the corresponding text-only and multimodal chat models, respectively.Source: Table 16 · p. 34, Table 17 · p. 35, Table 18 · p. 35

F.1 Cases for Error Analysis

The section presents error examples across disciplines and notes potential ethical and moral risks if AI performs increasingly well on the benchmark.

F.1 Cases for Error Analysis

G Consideration for Social Impact

H Limitations and Future Work

The benchmark may contain noisy problems, and its current scope is evaluating complex problem-solving rather than real-world task performance.

H Limitations and Future Work

Related papers

Loading 2406.12753v2…