PaperlayerView exact revision on arXiv

Source-linked AI summary

MorphoBench: A Benchmark with Difficulty Adaptive to Model Reasoning

Xukai Wang, Xuanbo Liu, Mingrui Chen, Haitian Zhong, Xuanlin Yang, Bohan Zeng, Jinbo Hu, Hao Liang, Junbo Niu, Xuchen Li, Ruitao Wu, Ruichuan An, Yang Shi, Liu Liu, Xu-Yao Zhang, Qiang Liu, Zhouchen Lin, Wentao Zhang, Bin Dong

arXiv:2510.14265v1cs.AI

TL;DR

Existing reasoning benchmarks are static and limited in scope, motivating evaluation that adjusts difficulty to model capability and spans multiple academic domains. MorphoBench combines multidisciplinary questions with adaptations to conditions and reasoning chains; results report that models perform better on easier adjusted questions and worse on harder ones.

  • Problem

    Existing reasoning benchmarks are limited in scope and do not adapt difficulty to changing model reasoning capabilities.

    Source: §1 · pp. 1–2

  • Method

    MorphoBench uses multidisciplinary reasoning questions and adjusts conditions and reasoning chains by modifying key statements in the model’s analysis.

    Source: §1 · p. 2

  • Results

    Across adjusted questions, models answer more accurately when questions become easier and less accurately as difficulty increases; o3 achieves 56.04% accuracy in social sciences.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    MorphoBench is designed to support fair, comparable reasoning evaluation across models with different proficiency levels.

    Source: §1 · p. 2

Abstract

from arXiv · show

With the advancement of powerful large-scale reasoning models, effectively evaluating the reasoning capabilities of these models has become increasingly important. However, existing benchmarks designed to assess the reasoning abilities of large models tend to be limited in scope and lack the flexibility to adapt their difficulty according to the evolving reasoning capacities of the models. To address this, we propose MorphoBench, a benchmark that incorporates multidisciplinary questions to evaluate the reasoning capabilities of large models and can adjust and update question difficulty based on the reasoning abilities of advanced models. Specifically, we curate the benchmark by selecting and collecting complex reasoning questions from existing benchmarks and sources such as Olympiad-level competitions. Additionally, MorphoBench adaptively modifies the analytical challenge of questions by leveraging key statements generated during the model's reasoning process. Furthermore, it includes questions generated using simulation software, enabling dynamic adjustment of benchmark difficulty with minimal resource consumption. We have gathered over 1,300 test questions and iteratively adjusted the difficulty of MorphoBench based on the reasoning capabilities of models such as o3 and GPT-5. MorphoBench enhances the comprehensiveness and validity of model reasoning evaluation, providing reliable guidance for improving both the reasoning abilities and scientific robustness of large models. The code has been released in https://github.com/OpenDCAI/MorphoBench.

1 Introduction

Existing reasoning benchmarks are static and often narrow in domain coverage, while some emphasize obscure knowledge over reasoning from familiar information. MorphoBench proposes a multidisciplinary benchmark with difficulty adjusted to model reasoning and a finer categorization of problem types.

1 Introduction

  • Existing reasoning benchmarks are static and often narrow in domain coverage, and some rely on obscure knowledge rather than complex inference from familiar information.The introduction calls for difficulty that adapts to model ability and coverage across academic domains.Source: §1 · pp. 1–2
  • MorphoBench combines multidisciplinary, reasoning-intensive problems with adaptive difficulty calibration to support evaluation across models of different reasoning abilities.The benchmark aims for fair and comparable evaluation across models with different proficiency levels.Source: §1 · p. 2
  • MorphoBench adjusts questions along condition recognition and reasoning-process dimensions by modifying key statements in the model’s analysis.The changes vary condition clarity and introduce guiding hints or distracting information to regulate reasoning complexity.Source: §1 · p. 2
  • The paper categorizes problem types and studies their frequencies to guide benchmark updates and increase its diversity.The authors describe this as supporting a fuller assessment of model abilities.Source: §1 · pp. 2–3

2 Related Work

Large-scale and multimodal models have advanced, increasing the need to evaluate their capabilities. Existing evaluation benchmarks cover multimodal, reasoning-focused, and specialized tasks, but cannot adapt to model reasoning performance.

2.1 Large Models

  • Large-scale and vision-language models have advanced, while evaluating large-model capabilities remains important.The related work describes advances in NLP, multimodal understanding, and reasoning.Source: §2.1 · p. 3

2.2 Evaluation Benchmark for Large Models

  • Existing benchmarks assess multimodal integration, complex reasoning, or specialized question answering, but cannot adapt to models’ reasoning performance.The resulting challenge is fair evaluation across models with varying capabilities.Source: §2.2 · p. 3

3 MORPHOBENCH

MORPHOBENCH combines multidisciplinary reasoning questions with a difficulty model based on proof-graph search. It adjusts problems through changes to reasoning steps, question cues, and generated-task parameters, while organizing coverage across task and skill categories.

3.1 Data Collection

  • Questions come from existing benchmarks, Olympiad competitions, and expert-designed scenarios whose answers are verified with simulation software.Selected sources include HLE, MME-Reasoning, HistBench, and competition problems in mathematics, physics, and chemistry.Source: §3.1 · p. 3

3.2 Preliminary Analysis

  • The analysis treats reasoning as a search process whose graph complexity, reflecting depth and branching, can be adjusted to control question difficulty.The proof graph represents intermediate reasoning statements and valid inference steps, with edge weights reflecting expected computational cost.Source: §3.2 · p. 4, §3.2.1 · p. 4

3.2.1 Reasoning as Path Search in a Proof Graph

  • A question’s intrinsic difficulty is the expected cost of deriving its answer across valid proof-graph paths, weighted by the model’s likelihood of following each path.The graph runs from the original problem statement to the fully verified answer.Source: §3.2.1 · p. 4

3.2.2 Question Modification and Information Gap

  • Modifications append hints, with positive information gaps indicating misleading or irrelevant adjustments that increase reasoning difficulty.Helpful or redundant modifications can reduce search depth by supplying intermediate constraints or decompositions.Source: §3.2.2 · p. 5

3.2.3 Impact of Modifications on Reasoning Complexity

  • For any fixed compute budget, misleading modifications increase the perturbed problem’s failure probability relative to the original.The analysis models misleading edges as added search cost along the reasoning path.Source: §3.2.3 · p. 5

3.3 Difficulty Adaptation

  • Reasoning-based adaptation adjusts hints and proof-graph structure, making steps more explicit or implicit, pruning lemmas, or extending lemma chains.Simple hints lower difficulty, while complex hints raise it.Source: §3.3 · p. 5
  • Recognition-based adaptation obfuscates model-identified critical visual cues, while generated tasks vary difficulty through exposed circuit terminals or visual similarity and grid size.Circuit simulators verify generated outputs; increasing terminal counts or grid complexity raises task difficulty.Source: §3.3 · pp. 5–6

3.4 Category Expansion

  • Problems are organized by task type, knowledge dependence, and fine-grained skill categories, with per-category quotas used to address sparse areas.The three task types are perception, retrieval, and reasoning; knowledge dependence is closed, open, or hybrid.Source: §3.4 · p. 6

4 Experiment

Experiments test MorphoBench across disciplines and progressively adjusted questions. Results show that model performance varies by domain and difficulty, while the benchmark’s taxonomy spans diverse reasoning tasks.

4.2 Main Comparison Results

  • Across adjusted versions, models answer easier questions more accurately and harder questions less accurately; GPT-5 degrades less than o3 on harder questions despite o3 leading on original items.Recognition-focused adjustments affect reasoning less than adjustments targeting reasoning capacity.Source: Table 2 · p. 7, §4.2 · pp. 7–8

4.1 Implementation Details

  • Models are assessed first on MORPHO-V0 and then on reasoning, recognition, and generated-circuit variants; accuracy is the proportion of correctly answered questions.The section reports discipline-level analysis across mathematics, engineering, natural sciences, and social sciences.Source: §4.1 · p. 7, §4.1.1 · p. 7

4.2 Main Comparison Results

  • 56.04% in social sciences, 53.26% in mathematics, and 41.75% in other tasks compare with 37.73% in engineering and 34.40% in natural sciences for o3.The passage associates stronger performance with textual representation and conceptual reasoning, and lower performance with symbolic derivation, quantitative manipulation, and expert-designed challenges.Source: §4.2 · p. 7
  • As difficulty rises from level 1 to 10, Gemini-2.5-Pro accuracy drops from 75.9% to 0–13%, while o3 fluctuates between 30% and 58% without a clear downward trend.Source: §4.2 · p. 8
  • Figure 5’s classification shows reasoning tasks predominate while all three top-level categories remain represented, spanning prompt-only and external-knowledge problems.Leaf categories range from combinatorics and geometry to timeline reasoning and logical entailment.Source: Figure 5 · p. 8, §4.2 · p. 8
  • After expansion and rebalancing, hierarchical evenness and entropy improve, and leaf coverage reaches approximately 60% of possible taxonomy paths.The authors identify Open/Hybrid knowledge closure, retrieval-anchored items, and perception tasks requiring open knowledge as priorities for future iterations.Source: §4.2 · p. 8

5 Conclusion

The conclusion presents MORPHOBENCH as a multidisciplinary reasoning benchmark whose difficulty can be adjusted to model ability. It also describes attribute balancing and experiments validating the benchmark’s design and utility.

5 Conclusion

  • MORPHOBENCH combines questions from multiple disciplines that demand strong reasoning with difficulty adjustment based on model reasoning ability.Adjustments add positive or negative guidance at analytical stages or modify the quality of critical information to recognize.Source: §5 · p. 8
  • The benchmark classifies question attributes in greater detail and balances them to improve diversity and comprehensiveness.Source: §5 · p. 8
  • The authors report conducting experiments to validate MORPHOBENCH’s design and utility.Source: §5 · p. 8

Appendices

The taxonomy labels each sample at three levels—task nature, knowledge closure, and reasoning primitive—and concatenates those labels into a leaf category. When labels are ambiguous, it prioritizes task nature, then knowledge closure, then the primary reasoning primitive.

A.1 Details of Taxonomy

  • L1 distinguishes perception or extraction, retrieval or matching, and reasoning or synthesis tasks.Reasoning or synthesis integrates evidence through multi-step deduction or constraint satisfaction.Source: §A.1 · p. 13
  • L2 marks whether a sample is closed, answerable from the prompt and evidence, or open, requiring external knowledge.Hybrid samples are primarily evidence-driven but may use a small amount of common or world knowledge to bridge gaps.Source: §A.1 · p. 13
  • L3 covers primitives including conservation, reachability, chronology, hierarchy, probability, arithmetic, counting, comparison, geometry, matching, and textual entailment.These categories encompass tasks such as checking logical consistency and applying balance principles.Source: §A.1 · p. 13
  • Each sample receives L1 task-nature, L2 knowledge-closure, and L3 reasoning-primitive labels, concatenated as its leaf category.Ambiguities are resolved in the order L1, then L2, then L3.Source: §A.1 · p. 13

A.2 More Proofs of Question Modification

The proof bounds how many distinct graphs can encode an incompressible string while preserving a unique goal path. The resulting lower bound is that the number of added dead-end edges must be at least the string length, up to a constant term.

A.2 More Proofs of Question Modification

  • m ≥ ΔI − O(1) added edges are required to embed an incompressible string of ΔI bits while preserving the unique goal path.The lower bound follows by combining the incompressibility assumption with a bound on the number of representable graphs.Source: §A.2 · pp. 13–14
  • A fixed computable embedding and constant-size decoding procedure let the graph description bound the encoded string’s Kolmogorov complexity.The construction starts with a single directed path and attaches dead-end edges without changing its unique goal route.Source: §A.2 · pp. 13–14
  • At most 2^m distinct graphs can result from adding m dead-end edges while preserving the path as the unique goal route.The proof counts choices of which outgoing edge continues along the path at each vertex.Source: §A.2 · p. 14

A.3 More Details of Difficulty Adjustment

The appendix describes difficulty adjustments driven by agent-identified visual cues, proof-graph reasoning, and automatically generated tasks. It also documents data sources and checks used in assembling the benchmark.

A.3 More Details of Difficulty Adjustment

  • Agent recognition perturbs key visual information identified by the model, making the same task test robustness to changes in cues it relies on.The authors contrast these targeted perturbations with externally imposed random noise.Source: §A.3 · p. 14
  • The proof graph represents reasoning depth and exploration breadth through lemmas and their directed inferential dependencies.Reasoning is modeled as progressively activating lemmas from axioms and previously established lemmas.Source: §A.3 · p. 15
  • Circuit questions use simulator-validated waveforms, with more exposed terminals increasing difficulty; visual odd-one-out grids become harder through greater similarity or larger size.The circuit structure is theoretically solvable given the component types.Source: §A.3 · p. 15
  • Increasing circuit terminal counts or visual-task grid size and similarity lets generated questions evolve toward harder problems as models advance.The authors present these mechanisms as supporting a dynamic, extensible evaluation platform.Source: §A.3 · p. 15

B More Information

  • Collected materials came from AoPS and Chinese mathematics and physics Olympiad training problems, with existing solutions or official answers and manual verification.The authors report obtaining permission from the data providers for research use.Source: B More Information · pp. 15–16
  • The authors report relying on established benchmark datasets without further anonymization checks and using AI only for grammar and minor style edits.They state that they concluded no additional data checks were necessary and that AI did not contribute to research design, analysis, or technical development.Source: B More Information · p. 16

C.1 Cross-disciplinary Analysis

Across five subject groups, O3 and GPT-5 have the most balanced and highest overall accuracies, while other models show disparities between formal and applied domains. The accompanying examples illustrate two difficulty-adjustment approaches: visual grounding and reasoning-chain interference.

C.1 Cross-disciplinary Analysis

  • O3 and GPT-5 achieve the most balanced and overall highest accuracies across five subject groups, while other models vary between formal and applied domains.Table 4 reports accuracy by aggregated subject category and weighted overall accuracy.Source: §C.1 · p. 16, Table 4 · p. 17
  • Grok-4 scores 49.11% in Mathematics but 5.47% in Engineering, a disparity the authors say lowers its overall weighted accuracy relative to top models.The authors associate the Engineering result with weaker generalization to applied physical reasoning and multi-step procedural understanding.Source: §C.1 · p. 16
  • GPT-5 records 57.53% accuracy in Mathematics and 52.75% in Social Sciences, while the authors report applied-reasoning limitations for Gemini-2.5-Pro and Claude-4.Their Engineering accuracies are reported as 7.73% and 37.58%, respectively.Source: §C.1 · pp. 16–17

C.2 Visualized Examples of Agent Recognition and Agent Reasoning Adjustments

  • Agent recognition weakens textual descriptions of model-identified visual cues, preserving solvability while increasing ambiguity and reliance on visual perception.The examples include symbols, numbers, geometric labels, and local regions.Source: §C.2 · p. 17
  • Agent reasoning adjustment adds irrelevant or partly related hints to test whether models distinguish essential reasoning steps from misleading information.The examples illustrate the two adjustment modes as complementary ways to reconfigure question difficulty.Source: §C.2 · pp. 17–18

D.1 Societal Impact

The authors report no direct negative societal impacts but caution about possible misuse. They also acknowledge that modifying questions does not yet generate entirely novel scientific reasoning problems.

D.1 Societal Impact

  • The benchmark is described as having no direct negative societal impacts, though the authors caution that unlawful individuals may misuse it.Source: §D.1 · p. 18

D.2 Future Work

  • Adapting existing questions to model reasoning still falls short of generating entirely novel scientific reasoning problems.Source: §D.2 · p. 18
  • The authors plan to use limitations observed in model reasoning to automate new question generation from reference literature.Source: §D.2 · p. 18

D.3 Potential Risks

The paper identifies privacy and exam-leakage risks and describes safeguards and data provenance. It also states that LLMs assisted only with language polishing and formatting, not annotation or conclusions.

D.3 Potential Risks

  • All benchmark items come from publicly available datasets, competitions, or exams with proper citations.Collection and annotation procedures are summarized in the main text, and sources are cited there or in the appendix.Source: §D.3 · p. 18
  • The dataset contains no personally identifiable information, and flagged sensitive content will be anonymized or removed.Source: §D.3 · p. 18
  • Some problems originate from past or mock exams and could be misused for drill purposes.Source: §D.3 · p. 18
  • LLMs were used only for language polishing and formatting, not answer annotation or drawing conclusions; human experts verified all labels.Source: §D.3 · p. 18

Related papers

Loading 2510.14265v1…