PaperlayerView exact revision on arXiv

Source-linked AI summary

SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines

P Team, Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, Chujie Zheng, Kaixin Deng, Shawn Gavin, Shian Jia, Sichao Jiang, Yiyan Liao, Rui Li, Qinrui Li, Sirun Li, Yizhi Li, Yunwen Li, David Ma, Yuansheng Ni, Haoran Que, Qiyao Wang, Zhoufutu Wen, Siwei Wu, Tyshawn Hsing, Ming Xu, Zhenzhu Yang, Zekun Moore Wang, Junting Zhou, Yuelin Bai, Xingyuan Bu, Chenglin Cai, Liang Chen, Yifan Chen, Chengtuo Cheng, Tianhao Cheng, Keyi Ding, Siming Huang, Yun Huang, Yaoru Li, Yizhe Li, Zhaoqun Li, Tianhao Liang, Chengdong Lin, Hongquan Lin, Yinghao Ma, Tianyang Pang, Zhongyuan Peng, Zifan Peng, Qige Qi, Shi Qiu, Xingwei Qu, Shanghaoran Quan, Yizhou Tan, Zili Wang, Chenqing Wang, Hao Wang, Yiya Wang, Yubo Wang, Jiajun Xu, Kexin Yang, Ruibin Yuan, Yuanhao Yue, Tianyang Zhan, Chun Zhang, Jinyang Zhang, Xiyue Zhang, Xingjian Zhang, Yue Zhang, Yongchi Zhao, Xiangyu Zheng, Chenghua Zhong, Yang Gao, Zhoujun Li, Dayiheng Liu, Qian Liu, Tianyu Liu, Shiwen Ni, Junran Peng, Yujia Qin, Wenbo Su, Guoyin Wang, Shi Wang, Jian Yang, Min Yang, Meng Cao, Xiang Yue, Zhaoxiang Zhang, Wangchunshu Zhou, Jiaheng Liu, Qunshu Lin, Wenhao Huang, Ge Zhang

arXiv:2502.14739v4cs.CL

TL;DR

Existing benchmarks emphasize common academic fields, while many long-tail professional disciplines remain inadequately evaluated. SuperGPQA evaluates graduate-level knowledge and reasoning across 285 disciplines through human–LLM collaboration. The leading reasoning models perform best, yet score only around 60.

  • Problem

    Existing benchmarks focus on common academic fields, leaving long-tail professional disciplines inadequately evaluated.

    Source: §1 · p. 4

  • Method

    SuperGPQA evaluates graduate-level knowledge and reasoning across 285 disciplines using a large-scale human–LLM collaboration system.

    Source: §1 · p. 4

  • Results

    Top reasoning models achieve the best overall performance on SuperGPQA, but leading systems score only around 60.

    Source: §1 · pp. 4–5

  • Takeaways & Limitations

    SuperGPQA provides a challenging frontier for evaluating LLMs across diverse graduate disciplines.

    Source: §1 · p. 4

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated remarkable proficiency in mainstream academic disciplines such as mathematics, physics, and computer science. However, human knowledge encompasses over 200 specialized disciplines, far exceeding the scope of existing benchmarks. The capabilities of LLMs in many of these specialized fields-particularly in light industry, agriculture, and service-oriented disciplines-remain inadequately evaluated. To address this gap, we present SuperGPQA, a comprehensive benchmark that evaluates graduate-level knowledge and reasoning capabilities across 285 disciplines. Our benchmark employs a novel Human-LLM collaborative filtering mechanism to eliminate trivial or ambiguous questions through iterative refinement based on both LLM responses and expert feedback. Our experimental results reveal significant room for improvement in the performance of current state-of-the-art LLMs across diverse knowledge domains (e.g., the reasoning-focused model DeepSeek-R1 achieved the highest accuracy of 61.82% on SuperGPQA), highlighting the considerable gap between current model capabilities and artificial general intelligence. Additionally, we present comprehensive insights from our management of a large-scale annotation process, involving over 80 expert annotators and an interactive Human-LLM collaborative system, offering valuable methodological guidance for future research initiatives of comparable scope.

1. Introduction

Existing benchmarks focus on common fields and have become less challenging, leaving long-tail professional knowledge undercovered. SuperGPQA addresses this with 285 graduate-level disciplines and a human–LLM collection and inspection process.

1. Introduction

  • SuperGPQA covers 285 graduate-level disciplines, with at least 50 questions per discipline for real-world professional accessibility.The benchmark is intended to probe knowledge and reasoning at the boundaries of human knowledge.Source: §1 · p. 4
  • Top reasoning models achieve the best overall SuperGPQA performance, but leading systems score only around 60.The introduction identifies reasoning models such as DeepSeek-R1 and o1 as top performers.Source: §1 · pp. 4–5
  • Transcription converts or translates raw questions into multiple-choice items, adds confusion options, and estimates difficulty and reliability using expert judgments and LLM responses.The process also uses plagiarism checks and SOTA-LLM filtering to reduce wasted annotation effort.Source: §1 · pp. 4–5
  • Quality inspection flags suspicious questions from LLM responses, sends them to experts for review, and further tailors easy items using model accuracy.Experts can use unrestricted web access when reviewing flagged questions.Source: §1 · p. 5
  • 47.40 and 40.75 for DeepSeek-V3 and Qwen2.5-72B-Instruct exceed 32.14 and 34.33 for DeepSeek-V3-Base and Qwen2.5-72B, respectively.The introduction presents these comparisons as evidence that instruction tuning is helpful.Source: §1 · p. 5
  • DeepSeek-R1 scores 63.59, 63.63, and 56.87 on simple, middle, and hard splits, while Qwen2.5-14B-Instruct scores 44.82, 37.90, and 19.97.The authors describe stronger models as producing more balanced results across difficulty levels.Source: §1 · p. 5

2. Data Collection

  • Experts source credible discipline-specific questions, which crowd annotators revise before a human–LLM inspection process selects difficult, reliable items.The collection pipeline has three stages: Source Screening, Transcription, and Quality Inspection.Source: §2 · pp. 5–6

4. Generated Distractors

The collection process combines expert-led sourcing and staged quality inspection to select reliable, difficult questions. The authors report that exercise-site answers can be erroneous and that providing source screenshots improves inspection efficiency.

R Answer R Difficulty

  • Generated distractors should be plausible but incorrect, subtly flawed, unique, and leave one correct answer.The instruction specifically frames this as distractor generation for a multiple-choice question.Source: R Answer R Difficulty · p. 6

2.1. Source Screening

  • Expert annotators collect questions from credible resources, while crowd-sourced questions were often judged too easy or unreliable by experts.The authors say this made some early annotation funding ineffective.Source: §2.1 · pp. 6–7
  • Exercise-site answers proved unreliable: correcting errors derived from them consumed substantial quality-inspection time, and several leading LLMs reproduced those errors.The authors report consistency by GPT-4o, o1-mini, and Gemini-flash with erroneous processes and answers from online exercise websites.Source: §2.1 · p. 7

solutions provided by online exercise websites is limited and there is a significant

  • Providing source screenshots eases quality inspection and improves its efficiency.Original-question selection prioritizes textbook examples with solutions, then reasoning or calculation questions from websites.Source: solutions provided by online exercise… · p. 7

B. Albinism

This section describes SuperGPQA’s question transcription and multi-stage quality inspection, and characterizes its 26,529 questions across 285 subfields in terms of disciplinary coverage, difficulty, reasoning demands, and semantic patterns. It also examines how the benchmark’s semantic structure reflects traditional academic organization and natural intellectual relationships, capturing domain-specific knowledge and cross-disciplinary connections.

2.2. Transcription

  • Transcription converts non-English and open-ended questions into academic English multiple-choice items, standardizes correct-or-incorrect formats, and adds regional context where needed.The authors note that crowdsourcing annotators have low accuracy in judging generated distractors, so flawed distractors in these question types require unified rewriting.Source: §2.2 · p. 8

2.3. Quality Inspection

  • Question quality inspection combines rule-based formatting checks, LLM-based validity and discrimination assessments, and expert review of suspicious candidates.Experts solve candidates with unrestricted web access and spend over 30 minutes on each, according to post-annotation interviews.Source: §2.3 · p. 9

3. Statistics

  • SuperGPQA contains 26,529 questions across 13 disciplines, 72 fields, and 285 subfields, with 9.67 options per question on average versus the conventional four.The paper says it exceeds GPQA (448 questions) and MMLU-Pro (12,032 questions) in question-pool size and has a more challenging format.Source: Table 1 · p. 10, §3 · p. 10
  • Science, Engineering, and Medicine account for 77.2% of questions, while smaller non-STEM subsets also discriminate among state-of-the-art models.The paper attributes the STEM-heavy distribution to more questions meeting stringent quality and difficulty criteria, and reports discrimination across domains regardless of sample size.Source: §3 · pp. 10, 13
  • 42.33% of questions require mathematical calculations or formal reasoning, with the highest reported calculation rates in Science (66.34%) and Engineering (55.40%).Difficulty profiles vary: Engineering is 31.1% hard, 43.9% middle, and 25.0% easy; Science is 42.8% hard, 42.0% middle, and 15.2% easy.Source: §3 · p. 13
  • Question-answer embeddings cluster by discipline while retaining overlap; Engineering and Science overlap most, whereas Military Science has relatively isolated patterns.The humanities show diffuse boundaries but distinct centroids, and the paper interprets Military Science’s isolation as reflecting domain-specific language.Source: §3 · p. 14
  • The benchmark statistics include correlations between benchmarks, with MMLU results recorded under the 5-shot setting.Figure 5 presents correlation coefficients between different benchmarks.Source: §3 · p. 15, Figure 5 · p. 15
  • SuperGPQA’s semantic structure reflects traditional academic organization and natural intellectual relationships, capturing domain-specific knowledge and cross-disciplinary connections.This conclusion is drawn from the discipline relationships shown in Figure 6.Source: §3 · p. 15, Figure 6 · p. 15

4. Experiments

The experiments compare reasoning, chat, and base models on SuperGPQA, with reasoning-focused models among the strongest overall. Results also distinguish knowledge-oriented performance from hard-question reasoning and examine prompt context and robustness.

4.1. Baseline Models

  • The evaluation covers 6 reasoning models, 28 chat models, and 17 base models; reasoning and chat models are zero-shot, while base models are five-shot.The five-shot base-model setup follows a methodology similar to MMLU-Pro.Source: §4.1 · p. 16

4.2. Main Results

  • Reasoning models including DeepSeek-R1 and o1-2024-12-17 achieve the best overall SuperGPQA performance.Table 5 reports performance at sample-wise, subfield-wise, field-wise, and discipline-wise levels.Source: §4.2 · p. 16, Table 5 · p. 17
  • DeepSeek-R1 beats DeepSeek-R1-Zero in only Science and Engineering, while R1-Zero is slightly better in other disciplines.The authors say this leaves the optimal training paradigm for reasoning models an open question.Source: Table 6 · p. 18, §4.2 · p. 19
  • The authors report that newer proprietary LLM versions improve on SuperGPQA and conjecture that developers value incorporating long-tailed knowledge.They partly attribute Chinese firms’ stronger performance to benchmark references translated from Chinese.Source: §4.2 · p. 19
  • Fully open-source models lag similarly sized alternatives, and most open-weight models aside from the DeepSeek-R1 series are unsatisfactory on the benchmark.The paper identifies pre-training corpus curation as one dimension of open-source progress.Source: §4.2 · p. 19

4.3. Further Analysis

  • Larger Qwen2.5-Instruct models benefit more from added subfield context and show greater stability, with lower variance across 24 semantically equivalent prompts.The analysis compares models from 0.5B to 72B parameters across prompts with and without subfield information.Source: §4.2 · p. 20, §4.3 · pp. 20–21

4.2. Main Results

  • Hard questions challenge reasoning, while easy and middle splits better reflect factual knowledge; o3-mini trails Doubao on the latter but surpasses it on hard questions.The authors present difficulty levels as diagnostic tools for distinguishing complementary model capabilities.Source: §4.2 · p. 20

show that incorporating subfield information generally leads to improved perfor-

Adding subfield context improves accuracy for larger Qwen2.5-Instruct models, while smaller models show little gain or a slight decline.

show that incorporating subfield information generally leads to improved perfor-

  • 41.93% with subfield information versus 40.82% without for Qwen2.5-72B-Instruct.Qwen2.5-32B-Instruct improves from 39.13% to 39.65%, and Qwen2.5-14B-Instruct from 35.36% to 35.78%.Source: show that incorporating subfield… · p. 21
  • Qwen2.5-0.5B-Instruct scores 10.62% with subfield annotations versus 11.12% without, while Qwen2.5-1.5B-Instruct is near-identical at 18.09% versus 18.62%.The authors suggest smaller models may rely more on general knowledge retrieval than contextual domain disambiguation.Source: show that incorporating subfield… · p. 21
  • The robustness test uses 24 semantically equivalent prompt styles formed from four initial prompts and six question formats.Prompt 1 paired with question format 1 is the default evaluation prompt.Source: show that incorporating subfield… · p. 21
  • Larger Qwen2.5-Instruct models exhibit increased robustness across the prompt variations.The supplied passage does not provide a numerical robustness comparison.Source: show that incorporating subfield… · p. 21

a steady improvement in accuracy from 9.80% of Qwen2.5-0.5B-Instruct to 41.15%

The ensembling analyses find that Best of N benefits Qwen2.5-72B-Instruct more, whereas majority voting favors Doubao-1.5-pro-32k-20250115.

Robustness analysis

  • Prompt-variation standard deviation falls from 1.37 for the smallest model to approximately 0.27–0.36 in the largest models.The analysis describes the evaluation framework as stable, with a maximum standard deviation of 1.37%.Source: a steady improvement in accuracy from… · p. 21

Best of N and Majority Voting

Best of N (BoN) Analysis

  • Qwen2.5-72B-Instruct has a steeper Best of N curve, while Doubao-1.5-pro-32k-20250115 starts stronger but gains more gradually.The authors suggest Qwen benefits more from multiple sampling attempts, while Doubao's responses are more consistent.Source: a steady improvement in accuracy from… · p. 22

32k-20250115 consistently outperforms Qwen2.5-72B-Instruct across all voting sizes,

Doubao-1.5-pro-32k-20250115 leads Qwen2.5-72B-Instruct across voting sizes, while disciplinary discrimination varies across domains.

32k-20250115 consistently outperforms Qwen2.5-72B-Instruct across all voting sizes,

4.4. Analysis of Disciplinary Discrimination Power

  • The discrimination analysis combines descriptive statistics with indices comparing the mean accuracy gap between each discipline's top three and bottom three models.The reported metrics include mean accuracy, standard deviation, coefficient of variation, and high-low group difference.Source: §4.4 · p. 23
  • Humanities show greater model-performance variation, whereas STEM performance converges; the authors attribute this contrast to differences in knowledge representation.They argue humanities better reveal capability differences beyond pattern recognition, while STEM metrics approach performance ceilings.Source: §4.4 · pp. 23–24, Table 7 · p. 24

5. Related Work

The related-work discussion situates SuperGPQA among advances in LLMs and benchmarks, then identifies the community and team contributions behind the work.

5.1. Large Language Models

  • SuperGPQA is presented as a benchmark designed to probe the knowledge boundaries of existing LLMs.Source: §5.1 · p. 25

5.2. LLM Benchmarks

  • Existing LLM benchmarks often focus on single tasks or domains, while SuperGPQA scales graduate-level evaluation to 285 disciplines.The paper also describes its contribution as fine-grained analysis of foundation models.Source: §5.2 · p. 25

6. Contributions and Acknowledgements

  • M-A-P is described as a donation-funded, nonprofit open-source AI research community; team contributions include data annotation management and quality inspection.Source: §6 · p. 26

A. Difficulty-Stratified Samples

This section presents difficulty-stratified sample questions and describes how materials are selected, transformed, and constructed as multiple-choice items. It also outlines the rule-based, LLM-assisted, and manual reviews used to check question and distractor quality, data integrity, and validity.

A. Difficulty-Stratified Samples

  • Sample questions span dance studies, a 52-card shuffle permutation problem, and free-space radio transmission calculations.The examples are tagged to Literature and Arts and Communication Principles; the excerpts do not report difficulty-level distributions or results.Source: Question · pp. 39–40

B.1. Material Requirements

  • Materials must be accurate, diverse, region-neutral, graduate-level and reasoning-oriented, non-image-based, and copyright-compliant.The requirements call for broad knowledge-point coverage and correctly stated material.Source: §B.1 · p. 41

B.2. Annotation Methods

  • The annotation process identifies original transcription, non-choice conversion, and statement combination among its methods.The overview says four methods are used, but the supplied passages describe only these three.Source: §B.2 · p. 42

B.2.1. Original Transcription Method

  • Original transcription preserves a question’s meaning and correctness while adding distractors to increase difficulty or discrimination.Annotators are also instructed to maintain accuracy, diversity, neutrality, non-image-based content, and copyright compliance.Source: Description · p. 42

B.2.2. Non-Choice Conversion Method

  • Non-choice questions are converted to multiple-choice format with complete, unambiguous conditions, accurate answers, and reasonable distractors.Distractors may reflect common calculation errors, with numerical options set to distinguish plausible error ranges.Source: Description · pp. 43–44

B.2.3. Statement Combination Method

  • Statement combination integrates multiple concepts into multi-level, multi-perspective questions using representative statements and plausible incorrect versions.Statements are drawn from textbooks, learning resources, and multiple-choice questions, and should cover multiple knowledge points without vague or obvious errors.Source: Description · p. 44
  • Statement-combination questions number and arrange adapted statements without directional bias, combine them into options, and require exactly one correct answer.Annotators also complete discipline, field, and subfield metadata and select a difficulty level.Source: Description · p. 45

B.2.4. Confusion-options Generation

  • Distractor options are generated by a randomly selected model from Claude-3.5, GPT-4, Doubao, and Qwen2.5-72B-Instruct, then double-checked by an annotator before use.Distractors should be plausible but incorrect, distinct from existing options, and preserve exactly one correct answer.Source: §B.2.4 · p. 45, Input Description · p. 46

C. Data Filtering and Manual Review Process Details

  • A three-stage workflow uses rule-based screening, LLM quality inspection, and final manual review to remove invalid data, identify errors, and correct misjudgments using domain knowledge.The stages are designed to improve efficiency, accuracy, completeness, and usability.Source: C. Data Filtering and Manual Review… · p. 47

C.1. Rule-Based Pre-Check

  • Rule-based pre-checks cover text normalization, question-option consistency, answer integrity, and completeness of difficulty, discipline, and other metadata.Table 8 is identified as the source of detailed pre-check criteria, but its specific contents are not supplied here.Source: §C.1 · p. 47

Category Rules

  • Category rules normalize punctuation and whitespace, check question text format and length, cap perplexity at 100, and screen for semantic duplicates.The stated perplexity model is Qwen2.5-0.5B-Instruct; duplicate search uses SentenceTransformer embeddings and Faiss.Source: Category Rules · p. 48

C.2. LLM-Based Quality Inspection

  • LLM-based inspection covers validity, negative or extreme inquiries, multimodal exclusions, field relevance, and solvability.Source: §C.2 · p. 48

C.2.1. Validity Check

  • Validity checks require a clear, solvable multiple-choice question with complete, relevant options and an answer, while rejecting negative phrasing and open-ended formats.Options must stand independently rather than refer to one another.Source: §C.2.1 · p. 49, answer: "Answer content" · pp. 49–50

question: "{}",

This section describes criteria and review checks for identifying problematic questions, including issues with wording, image dependence, field classification, answerability, ambiguity, missing context, and incomplete construction. Together, these checks guide judgments about whether questions are suitable and reliably answerable.

C.2.2. Negative and Extreme Inquiry Detection

  • Questions with negation or “most” wording are flagged, while vague best/worst phrasing and all-affirmation or all-negation choices are treated as unsuitable.The criteria target negative or extreme inquiries because distractors may satisfy their conditions.Source: §C.2.2 · pp. 50–51

C.2.3. Multimodal Exclusion

  • A question is excluded for multimodal dependence only when it cannot be solved without visual input; text-solvable questions remain eligible.The prompt distinguishes strictly image-required questions from those answerable through text, including math problems, logic puzzles, and verbal descriptions.Source: §C.2.3 · p. 54

C.2.4. Field Relevance Evaluation

  • Classification is assessed from discipline to field to subfield, and a highly relevant subfield can validate the full classification despite minor mismatches above it.If no level meets the criteria, or an upper level is inappropriate without a highly relevant subfield, the classification is not relevant.Source: §C.2.4 · pp. 54–55
  • A classification is deemed not relevant when no level meets the criteria or an upper level is inappropriate without a highly relevant subfield.This is the fallback to the level-by-level assessment.Source: §C.2.4 · p. 55

C.2.5. Completeness Assessment

  • Completeness assessment separates missing necessary information from difficulty: absent diagrams or critical conditions make a problem unsolvable, while difficulty affects confidence.The prompt assigns confidence as High, Medium, or Low based on simplicity, number of steps, complexity, and uncertainty.Source: §C.2.5 · p. 56
  • When a necessary diagram is absent, mark missing_info true; when the problem has sufficient information but is difficult, mark missing_info false and set confidence from the reasoning.The rubric also treats missing numbers, formulas, or critical conditions as missing information.Source: Rules · p. 56

C.3. Manual Quality Review

  • Manual review checks consistency, clarity, accuracy, global perspective, appropriate difficulty, answer correctness and uniqueness, and proper categorization.Table 9 is identified as the detailed guideline for this review.Source: §C.3 · p. 58

C.4. Reasons for Failing Quality Inspection

  • Examples flag unsolvable items when essential conditions are absent, including a required diagram or the iron cube’s mass.The resistor problem cannot be solved without its diagram, and the heat-transfer question omits the cube’s mass.Source: §C.4 · pp. 59, 62
  • Reviewers identify incorrect option construction when multiple choices are correct and distractor quality issues when options are irrelevant or poorly constructed.The gastrointestinal-function example has several correct choices, while another item uses distractors such as “infinity” and “undefined.”Source: §C.4 · pp. 60–61
  • Examples also flag missing context: a Ross question lacks background or theory, while a China-specific item needs region-aware selection or modification.The latter could also be refined to improve clarity and relevance.Source: §C.4 · pp. 60, 62
  • Other failures include a voice-leading question without its required example and a multi-part question whose answer and options address only one sub-question.The absent notation prevents the example-based item from being answered, while the incomplete answer construction causes ambiguity.Source: §C.4 · p. 63

D. Quantitative Overview of Disciplines at Three Levels

This section surveys discipline coverage at three levels across a broad range of fields and describes prompt designs for evaluating subfield information and robustness.

D. Quantitative Overview of Disciplines at Three Levels

F.2. Five-shot Prompt

  • The five-shot prompt includes a worked short-period-comet example that reasons from orbital alignment to the Kuiper belt.The example links comets’ orbits in the solar-system plane to the Kuiper belt’s distribution.Source: §F.2 · pp. 70–71

G.1. Impact of Subfield Information

  • The subfield-information evaluation compares zero-shot prompts that include a problem’s subfield description with prompts that omit it.The supplied passages define these two conditions but do not report their results.Source: §G.1 · p. 72

G.2. Robustness of the Evaluation

  • The robustness check combines four initial prompts with six question formats to create 24 prompt styles.The passage describes this setup as a way to verify the benchmark’s robustness.Source: §G.2 · p. 73

I. The difference in the common-mode rejection ratio of A1 and A2 themselves

This material presents a common-mode rejection question and its answer, lists benchmarks used for data supplementation, and describes broader baseline-performance tables. The tables cover multiple scoring levels, difficulty bands, and disciplines.

I. The difference in the common-mode rejection ratio of A1 and A2 themselves

  • The first-stage common-mode rejection ratio question is answered as the absolute difference between A1 and A2’s common-mode rejection ratios.The question concerns a three-op-amp differential circuit.Source: Question · p. 76

H. Benchmarks for Data Expansion

I. More Comprehensive Analysis of Baseline Performances

  • Tables 12 and 13 provide broader baseline-performance overviews, covering scoring by sample, subfield, field, and discipline, difficulty bands, and discipline-specific performance.The accompanying text names models from Yi-1.5, OLMo-2, Mistral, granite-3.1, Gemma-2, Llama-3.1, Mixtral, and K2.Source: §I · p. 78, Table 12 · p. 79, Table 13 · p. 80

Architectural History Contract Law Botany

The listed disciplines span engineering, law, humanities, and the natural sciences, with examples ranging from architectural and legal subjects to biology and chemistry.

Architectural History Contract Law Botany

  • Listed examples include Urban Planning and Design, Geotechnical Engineering, and Chemical Transport Engineering.The same list also names Structural Engineering and Urban Infrastructure Engineering.Source: Architectural History Contract Law… · p. 86
  • The discipline list includes Criminal Law, Political Science, International Law, and Legal Theory and Legal History.Other listed areas include Broadcasting and Television Art and Classical Chinese Literature.Source: Architectural History Contract Law… · p. 86
  • Natural-science examples in the list include Cell Biology, Analytical Chemistry, Ecology, Genetics, Microbiology, and Zoology.The list also names Elements of Chemical Reaction and Mass Transport and Separation.Source: Architectural History Contract Law… · p. 86

List of Model Score Tables

This section catalogs model-score tables reporting discipline-level performance for a range of language models across three levels of disciplines. The supplied entries span varied fields and models.

List of Model Score Tables

  • Tables 21–23, 26, 29, and 31 report model scores across three discipline levels for Doubao-1.5-pro-32k-241225, qwen-max-2025-01-25, claude-3-5-sonnet-20241022, DeepSeek-V3, gpt-4o-2024-11-20, and Llama-3.1-405B-Instruct.Source: Table 21 · p. 112, Table 22 · p. 114, Table 23 · p. 116
  • 78.00 is reported for Information Management and Communication and Radiation Medicine, while Modern and Contemporary Chinese Literature is listed at 26.00.These are discipline-level scores in the supplied entries.Source: List of Model Score Tables · p. 133

List of Model Score Tables

  • The listings cover arts and humanities disciplines, including Modern and Contemporary Chinese Literature (24.00), Russian Language and Literature (24.56), and Musical Forms and Analysis (28.00).Source: List of Model Score Tables · p. 233
  • They also report scores in medicine and science, including Radiation Medicine (32.00), Marine Biology (36.00), and Geochemistry (30.00).Source: List of Model Score Tables · p. 233
  • Scores range from 5.45 for Pharmacology to 44.00 for Information Management and Communication among the displayed entries.Other high entries include Human Geography (42.00) and Hydrogeology (40.00).Source: List of Model Score Tables · p. 233
  • Tables 88 and 89 present scores across three discipline levels for Qwen2.5-0.5B-Instruct and Qwen2.5-0.5B, respectively.Each table continues in a subsequent table; the supplied captions do not include score values.Source: Table 88 · pp. 246–247, Table 89 · pp. 248–249

Related papers

Loading 2502.14739v4…