Source-linked AI summary
OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, Maosong Sun
TL;DR
As established benchmarks become less challenging and often omit multimodal scientific reasoning, the paper introduces OlympiadBench, a bilingual benchmark of difficult mathematics and physics problems with expert annotations. It evaluates leading models and finds GPT-4V reaches only 17.97% overall, with particularly low physics performance, while analyses identify recurring reasoning failures.
Problem
Existing benchmarks are becoming insufficiently challenging and often lack multimodal coverage for evaluating advanced mathematical and scientific reasoning.
Method
OlympiadBench assembles 8,476 bilingual mathematics and physics problems from Olympiad and Chinese examination sources, with expert step-by-step annotations and comprehensive evaluation.
Results
17.97% is GPT-4V’s average score on OlympiadBench, including 21.70% in mathematics and 10.74% in physics.
Takeaways & Limitations
OlympiadBench provides a challenging benchmark for comparing large models’ mathematics and physics reasoning and analyzing their prevalent errors.
Takeaways & Limitations
Some proof-like and categorical answers require manual review because regular expressions and code-level tools cannot assess them reliably.
Abstract
from arXiv · showhide
Recent advancements have seen Large Language Models (LLMs) and Large Multimodal Models (LMMs) surpassing general human capabilities in various tasks, approaching the proficiency level of human experts across multiple domains. With traditional benchmarks becoming less challenging for these models, new rigorous challenges are essential to gauge their advanced abilities. In this work, we present OlympiadBench, an Olympiad-level bilingual multimodal scientific benchmark, featuring 8,476 problems from Olympiad-level mathematics and physics competitions, including the Chinese college entrance exam. Each problem is detailed with expert-level annotations for step-by-step reasoning. Evaluating top-tier models on OlympiadBench, we implement a comprehensive assessment methodology to accurately evaluate model responses. Notably, the best-performing model, GPT-4V, attains an average score of 17.97% on OlympiadBench, with a mere 10.74% in physics, highlighting the benchmark rigor and the intricacy of physical reasoning. Our analysis orienting GPT-4V points out prevalent issues with hallucinations, knowledge omissions, and logical fallacies. We hope that our challenging benchmark can serve as a valuable resource for helping future AGI research endeavors. The data and evaluation code are available at \url{https://github.com/OpenBMB/OlympiadBench}
1 Introduction
Existing benchmarks are becoming insufficient for evaluating advanced mathematical and multimodal scientific reasoning. OlympiadBench addresses this gap with a challenging bilingual benchmark and shows that leading models still perform poorly, especially in physics.
- 97.0% on GSM8K and 84.3% on MATH are GPT-4’s reported success rates, motivating more difficult benchmarks as these datasets approach saturation.
- Text-focused benchmarks inadequately test multimodal scientific reasoning required for contexts such as geometry and experimental design.
- 8,476 mathematics and physics problems from international and Chinese competitions and challenging GaoKao segments form OlympiadBench.
- The dataset combines bilingual multimodal problems with expert-level solution annotations, detailed labels, broad modality coverage, and varied answer formats.
- 17.97% is GPT-4V’s average OlympiadBench score, with 21.70% in mathematics and 10.74% in physics.
- Model analyses reveal computational errors, incorrect reasoning, induction failures, hallucinated reasoning, and unnecessarily complex solutions.
2 Related Work
Prior mathematics, physics, and multimodal benchmarks often limit difficulty, modality, question format, or scope. OlympiadBench is positioned as a broader and more challenging benchmark across these dimensions.
- Mathematics Benchmarks: GSM8K mainly targets elementary text-only arithmetic, while competition-level datasets are increasingly being surpassed by advanced models.
- Physics Benchmarks: Existing physics benchmarks commonly use elementary or high-school multiple-choice questions, while text-only multistep datasets remain limited in scope.
- Physics Benchmarks: SciBench offers college-level free-response physics with multimodal information, but OlympiadBench increases difficulty, question-type diversity, and volume.
- Multimodal Benchmarks: Multimodal benchmarks cover geometry and broad scientific disciplines, but the paper identifies insufficient benchmarks for multimodal scientific reasoning at Olympiad level.
3 The OlympiadBench Dataset
OlympiadBench is designed as a competition-level bilingual multimodal dataset emphasizing difficult open-ended scientific reasoning, detailed solutions, diverse subfields, and structured evaluation.
- Design Principle: OlympiadBench targets the highest competition level and includes open-ended mathematics and physics problems from international, regional, national, and GaoKao sources.
- Design Principle: Each problem receives expert-crafted solutions that expose reasoning steps and support annotation, evaluation, and research on model reasoning.
- Design Principle: Visual problems test image interpretation, spatial relationships, and visual information as components of scientific reasoning.
- Data Processing: Official competition PDFs are converted to Markdown, manually verified, structured as problem-solution-answer records, and deduplicated to reduce leakage risks.
- Data Characteristics: The dataset covers open-ended and theorem-proving questions across broad mathematical and physical subfields with manual topic and problem-type labels.
- Data Characteristics: Progressive physics problems link later questions to earlier answers or information, while open-ended answers are categorized into numeric, expression, equation, interval, and tuple types.
- Evaluation: The automated scoring pipeline handles numeric values with floating-point checks and symbolic expressions with symbolic computation, including tolerance-based physics evaluation.
4 Experiments
The experiments evaluate bilingual LMMs and text-only LLMs on OlympiadBench under standardized zero-shot prompting and automatic scoring. Results show the benchmark is highly challenging, with substantial model gaps and especially difficult multimodal physics and non-English questions.
- Main results: OlympiadBench widens performance gaps between models and enables more discriminating comparisons of their capabilities.The authors report that its difficulty provides a new perspective for comparing LMMs.
- Model comparisons: GPT-4V’s average accuracy was more than five times that of the best-performing open-source model, Yi-VL-34B.The authors also report that Gemini-Pro-Vision performed only slightly above an open-source model on these complicated tasks.
- Difficulty patterns: Image-based questions, physics, and non-English text were more challenging, with physics questions requiring physical laws and world knowledge beyond calculation and reasoning.Physics questions with images were especially difficult compared with mathematics questions.
- Model comparisons: DeepSeekMath-7B-RL outperformed or matched Gemini-Pro-Vision and Qwen-VL-Max on text-only questions, especially mathematics problems.This result was observed despite DeepSeekMath-7B-RL’s relatively small size.
- Model comparisons: GPT-4 performed slightly better than GPT-4V on all text-only OlympiadBench datasets except En_COMP.The authors hypothesize that the En_COMP difference may reflect improved long-context text reasoning.
5 Analysis
The analysis examines GPT-4V’s theorem-proving answers and sampled failures to characterize recurring reasoning problems. It finds weak proof performance alongside errors involving images, algebraic transformations, classification, concepts, and large calculations.
- Theorem proving: GPT-4V correctly answered only 6 of 81 sampled Math-Zh_COMP theorem-proving questions, mainly simple or computational ones.It was basically unable to complete proofs in Math-En_COMP within the token limit.
- Theorem proving: GPT-4V often failed to use image information, simplify algebraic expressions, avoid basic false conclusions, and handle classification discussions in proof problems.These issues were identified through manual examination of theorem-proving answers.
- Mistake analysis: 164 sampled failed open-ended problems were analyzed to characterize the distribution of errors in GPT-4V’s solving process.The sample included 97 mathematics and 67 physics Olympiad-level problems.
- Mistake analysis: Mathematics errors commonly involved insufficient classification discussion and poor logic during large calculations, while quadratic equations and derivatives were relative strengths.Conic-curve problems were cited as an example of difficult large-calculation tasks.
6 Discussion and Future Work
The discussion identifies theorem-proof evaluation and disciplinary coverage as important future-work boundaries. Automatic natural-language proof evaluation remains difficult, while the benchmark currently focuses on mathematics and physics.
- Evaluation of theorem proofs: Automatic evaluation of theorem proofs remains challenging because mainstream proof evaluation requires formalization, domain expertise, and background knowledge.The dataset’s natural-language and LaTeX annotations provide a resource for developing proof-evaluation methods.
- Expansion of disciplines: OlympiadBench currently focuses on mathematics and physics, with future work proposing expansion to geography, biology, and chemistry.The authors frame broader disciplinary coverage as a way to provide more comprehensive evaluation of reasoning capabilities.
7 Conclusion
The paper concludes by introducing OlympiadBench as a challenging bilingual multimodal benchmark for mathematical and physical reasoning. Its expert annotations and model-error analysis are intended to support future AGI and scientific-reasoning research.
- Conclusion: OlympiadBench is a bilingual multimodal benchmark designed to assess large-model reasoning in mathematics and physics.Each problem includes expert-level annotations for step-by-step reasoning.
- Conclusion: The benchmark analyzes model performance and prevalent error types while providing a resource intended to advance AGI and scientific-reasoning research.The authors state that future efforts will gather more challenging questions and broaden the benchmark’s scope.
Ethical Considerations
OlympiadBench is intended as a rigorous bilingual multimodal resource for evaluating scientific reasoning and supporting future AGI research. Its construction includes detailed annotations and publicly released data and evaluation scripts.
- OlympiadBench targets mathematical and physical reasoning in bilingual, multimodal settings.
- Each problem receives expert-level reasoning annotations to support detailed evaluation of model responses.
- The authors plan to release the dataset and accompanying evaluation scripts publicly to reduce unnecessary carbon footprint.
Limitations
OlympiadBench’s evaluation has important constraints for open-ended answers, automated scoring, and dataset development. These limitations require manual review and restrict the diversity and difficulty of multimodal scientific challenges.
- Proofs and categorical or textual answers cannot currently be assessed with regular expressions or SymPy and require manual review.
- The automated scoring system makes judgments from entered symbols or numerical expressions without incorporating problem-specific constraints.
- Extensive manual effort in gathering and annotating multimodal scientific data constrains the diversity and difficulty of available challenges.
A.1 Data Sources
OlympiadBench combines global, Chinese, and Gaokao mathematics and physics problems to create a multilingual dataset spanning advanced reasoning demands and a smooth difficulty transition. Its construction includes competition selection, annotation, model coverage, and multimodal problem handling.
- Global Mathematics and Physics Olympiad Problems: Global Olympiads contribute complex mathematics and physics problems requiring multiple solution methods and cross-subfield integration.
- Regional and National Chinese Mathematics Competitions: Chinese regional and national competitions add Chinese-context problems to support Chinese-oriented and multilingual model development.
- Gaokao Mock Questions for Mathematics and Physics: Gaokao simulation problems extend evaluation to smaller models answering free-form mathematics and physics questions.
- Combining Gaokao, Chinese competitions, and global Olympiads creates a smooth difficulty transition for distinguishing model capabilities.
- The dataset process surveys competitions by difficulty, volume, public availability, language, discipline, and coverage years before selecting contests and years.
- Physics problems may be progressive, with later questions depending explicitly or implicitly on earlier solutions and information.
- Evaluation includes current bilingual LMMs and LLMs across open- and closed-source models, using released checkpoints or official APIs.
- For text-only questions, the study uses corresponding text-model APIs or base LLMs instead of requiring image inputs for selected multimodal models.
B.2 Detailed Experiment Result
The detailed experiments compare mainstream models across benchmarks and examine GPT-4V’s performance by context length and knowledge point. Results also document unavailable responses and show particularly weak performance in several mathematics and physics areas.
- Long-PhO questions form relational series with long context, unlike normal-PhO questions.
- GPT-4 performs slightly better than GPT-4V on normal-PhO questions, while GPT-4V lags much behind on long-PhO questions.The comparison separates ordinary questions from relational series with long context.
- Some problem responses are unavailable because inputs exceed API limits, trigger inappropriate-response bans, return no response, or time out.
- Unavailable-response problems are removed when calculating accuracy.
- GPT-4V performs poorly on geometry, computationally intensive conic curves, and combinatorial classification discussions, but is stronger on derivatives and complex numbers.
- No physics knowledge point exceeds 16% accuracy, with greater difficulty in thermodynamics and mechanics.
C.2 Detailed Description of the Error Types in GPT-4V’s solving or proving process
GPT-4V’s solving and proving failures span misunderstanding, arithmetic and algebra errors, conceptual and logical mistakes, incomplete case analysis, hallucinated conclusions, and inefficient or image-insensitive reasoning.
- Basic and conceptual errors: GPT-4V sometimes misunderstands question settings, makes value or expression calculation errors, and produces false reasoning or conceptual confusion.These errors include simple arithmetic mistakes, incorrect transformations between expressions, and mistaken interpretations of concepts.
- Reasoning-process errors: GPT-4V may introduce unnecessary variables or concepts that lengthen solutions, create confusion, and lead to incorrect outputs.The issue concerns additions that contribute nothing to solving the problem.
- Reasoning-process errors: GPT-4V sometimes hallucinates unsupported conclusions or nonexistent theorems, causing subsequent proofs to lose their logic.The paper gives the nonexistent “Power Theorem” as an example in geometric proofs.
- Open-ended solution behavior: GPT-4V may produce unfinished answers, overly complex solutions, or correct answers supported by false processes.It may also identify the overall idea correctly but fail during calculation, yielding a wrong final answer.
- Incomplete case analysis: GPT-4V can omit or overlap cases in classification discussions, especially in combinatorial problems.The paper illustrates this failure with a die problem where the model incorrectly classifies an even number as odd.
- Multimodal reasoning errors: GPT-4V may fail to use image information, including misreading geometric relationships in figures.The paper also reports confusion between “vertical to the plane” and “vertical to the ground.”