PaperlayerView exact revision on arXiv

Source-linked AI summary

MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains

Guoli Yin, Haoping Bai, Shuang Ma, Feng Nan, Yanchao Sun, Zhaoyang Xu, Shen Ma, Jiarui Lu, Xiang Kong, Aonan Zhang, Dian Ang Yap, Yizhe zhang, Karsten Ahnert, Vik Kamath, Mathias Berglund, Dominic Walsh, Tobias Gindele, Juergen Wiest, Zhengfeng Lai, Xiaoming Wang, Jiulong Shan, Meng Cao, Ruoming Pang, Zirui Wang

arXiv:2407.18961v3cs.AI

TL;DR

Existing agent benchmarks often emphasize task completion without disentangling the capabilities behind outcomes, and interactive setups can be costly or unstable. MMAU evaluates five capabilities across five domains with static tasks. Its analyses find differences in capability difficulty, including challenges with self-correction and planning.

  • Problem

    Existing agent benchmarks emphasize application-specific task completion, obscuring which underlying capability fails, while environment setup can be costly and interactive results unstable.

    Source: §1 · p. 2

  • Method

    MMAU uses 20 tasks across five domains and five capabilities, with 3,220 prompts evaluated on a static dataset.

    Source: §1 · p. 2

  • Results

    Capability difficulty varies: problem-solving is more universally achievable, self-correction and planning are harder, and high-quality planning raises Command R’s Math performance from 8.21% to 33.33%.

    Source: §5 · p. 9

  • Takeaways & Limitations

    MMAU provides a test bed for more granular insight into LLM agents’ strengths and limitations.

    Source: §6 · p. 9

    MMAU omits interactive environments and its capability decomposition still faces challenges disentangling compound capabilities.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recent advances in large language models (LLMs) have increased the demand for comprehensive benchmarks to evaluate their capabilities as human-like agents. Existing benchmarks, while useful, often focus on specific application scenarios, emphasizing task completion but failing to dissect the underlying skills that drive these outcomes. This lack of granularity makes it difficult to deeply discern where failures stem from. Additionally, setting up these environments requires considerable effort, and issues of unreliability and reproducibility sometimes arise, especially in interactive tasks. To address these limitations, we introduce the Massive Multitask Agent Understanding (MMAU) benchmark, featuring comprehensive offline tasks that eliminate the need for complex environment setups. It evaluates models across five domains, including Tool-use, Directed Acyclic Graph (DAG) QA, Data Science and Machine Learning coding, Contest-level programming and Mathematics, and covers five essential capabilities: Understanding, Reasoning, Planning, Problem-solving, and Self-correction. With a total of 20 meticulously designed tasks encompassing over 3K distinct prompts, MMAU provides a comprehensive framework for evaluating the strengths and limitations of LLM agents. By testing 18 representative models on MMAU, we provide deep and insightful analyses. Ultimately, MMAU not only sheds light on the capabilities and limitations of LLM agents but also enhances the interpretability of their performance. Datasets and evaluation scripts of MMAU are released at https://github.com/apple/axlearn/tree/main/docs/research/mmau.

1 Introduction

MMAU addresses benchmarks that obscure which capabilities underlie agent task outcomes and can require costly, unstable interactive setups. It evaluates agents across capabilities and domains using a static dataset, while complementing rather than replacing interactive evaluation.

1 Introduction

  • Existing benchmarks emphasize application-specific task completion [9] [10] [11] [12], making it difficult to distinguish comprehension, reasoning, or calculation failures.Interactive evaluations can also require considerable setup and produce less stable, reproducible results because of stochastic environment feedback.Source: §1 · p. 2
  • MMAU evaluates five capabilities across five domains in 20 tasks, using 3,220 prompts performed on a static dataset.The capabilities are Understanding, Reasoning, Planning, Problem-solving, and Self-correction; the domains are Tool-use, DAG QA, Data Science and Machine Learning coding, Contest-level programming, and Mathematics.Source: §1 · p. 2
  • MMAU evaluates 18 commercial and open-source models and analyzes both application-specific performance and variation across capabilities.The authors present the benchmark as a way to gain deeper insight into agent strengths and limitations.Source: §1 · p. 3
  • The unified static-dataset evaluation is intended to avoid interactive-evaluation instability, and the dataset and evaluation scripts are released.The authors state that MMAU complements rather than replaces interactive evaluation, which they consider necessary.Source: §1 · p. 3

2 Related Work

Prior work includes generalist-agent frameworks and benchmarks that combine or target particular application domains. MMAU instead centers evaluation on core agent capabilities and aims to measure them in a disentangled manner.

2 Related Work

  • Generalist-agent work ranges from Auto-GPT’s tool-assisted goal decomposition to multi-agent collaboration frameworks [17] [18] [20].These frameworks explore collaboration, practical applications, control and customization, specialized-agent generation, and complex-task management.Source: §2 · p. 3
  • AgentBench [12] and AgentBoard [11] broaden benchmark coverage, but require containerized environments and substantial effort to implement new tasks.The authors argue that simply stacking tasks yields diminishing returns in comprehensiveness.Source: §2 · p. 3
  • MMAU focuses on core agent capabilities and tasks designed to produce decoupled metrics; Table 1 compares benchmarks’ capability support and entangled versus disentangled evaluation.The authors position MMAU as offering more comprehensive evaluation of fundamental capabilities in a disentangled manner.Source: §2 · p. 4, Table 1 · p. 4

3 The MMAU Benchmark

MMAU separates core agent capabilities from end-to-end task outcomes and evaluates them across diverse domains. Its tasks use curated data sources and evaluation setups designed to isolate particular skills.

3.1 Capabilities in MMAU

  • MMAU assesses understanding through complex instruction following, user-intent understanding, statistics parsing, and visual grounding.The benchmark also identifies reasoning, planning, problem-solving, and self-correction as essential capabilities.Source: §3.1 · p. 4
  • Planner-shift varies the planner while holding the solver fixed; solver-shift varies the solver while holding the planner fixed.The two-stage setup separates reasoning and planning from problem-solving.Source: §3.1 · p. 4, Figure 4 · p. 5

3.2 Dataset Construction

  • MMAU draws on in-house tool-use data, Kaggle, CodeContest [13], and DeepMind-math [15], with data curated or rewritten for its tasks.The tool-use data includes human-authored scenarios, ground-truth calls, and actual RapidAPI returns.Source: §3.2 · p. 5, §3.2.1 · p. 5

3.2.1 Tool-Use

  • Tool-use evaluation checks predicted calls against ground truth using prior turns, while DAG QA asks models to select and order dependent tool invocations.Tool-use self-correction introduces temporary errors or mutates calls, then evaluates whether the model retries correctly.Source: §3.2.1 · p. 5

3.1 Capabilities in MMAU

  • Tool-use self-correction tests retries after temporary tool failures or incorrect calls, scoring exact matches to the ground-truth function and arguments.The broader capability is defined as identifying errors and correcting behavior to overcome obstacles.Source: §3.1 · pp. 4–5, §3.2.1 · p. 5

3.2.2 Data Science and Machine Learning

  • The DS&ML dataset contains 28 notebook-style conversations and uses execution-grounded questions to assess generated text and image outputs.Settings evaluate end-to-end coding and QA, code generation with GPT-4 QA, QA from oracle code, and self-correction after execution errors.Source: §3.2.2 · p. 6

3.2.3 Contest-Level Coding

  • Contest-level coding uses 261 CodeContests problems; tasks include pass@K coding, planner- and solver-shift, problem parsing, and feedback-based self-correction.Problem parsing requires predicting test-case outputs without writing or executing code.Source: §3.2.3 · pp. 6–7

3.2.4 Mathematics

  • The math domain uses 1,000 problems spanning 56 subjects and includes end-to-end, planner-shift, solver-shift, and Comprehend+ tasks.Comprehend+ rewrites mathematically simpler problems with more complex descriptions while retaining their underlying mathematical constructs; it contains 676 problems.Source: §3.2.4 · p. 7

4 Evaluation

MMAU evaluates 18 models using domain-centric and capability-centric comparisons. The results show a commercial/open-source performance gap across domains and distinct patterns across capabilities.

4 Evaluation

  • The evaluation covers 18 models, and capability scores aggregate mapped tasks using weighted averages.The main paper presents aggregated results; detailed results for all 20 tasks are in Appendix C.Source: §4 · p. 7, §4.2 · p. 8

4.1 Domain-centric Evaluation and Analysis

  • A clear performance gap separates API-based commercial and open-source models across all evaluation domains, and the GPT-4 family consistently outperforms other models.GPT-4o has a significant advantage in math and contest-level coding; among open models, strengths vary by domain and tool-use support.Source: §4.1 · pp. 7–8

4.2 Capability-centric Evaluation and Analysis

  • GPT-4o leads on understanding, the GPT-4 family leads on reasoning and planning, problem-solving scores show a smaller gap, and self-correction remains uneven.Among open-source models, Mixtral-8x22B is the exception to the reported weakness in effective self-correction.Source: §4.2 · p. 8, Table 3 · p. 8

5 Analysis and discussion

MMAU’s analysis finds that planning quality can improve mathematical problem-solving, while capability difficulty and model scaling vary across models. Strong models show balanced capability profiles, but self-correction remains challenging for many open-source models.

Evaluation scope

  • Documented prompt templates for the evaluated models do not support multi-tool execution, although adaptation may make it possible.This qualification concerns current reported templates, not whether models can ever execute multiple tools.Source: §5 · p. 8

Planning and performance

  • High-quality planning improves Math performance across models, including GPT-4o from 53.4% to 61.2%.Command R rises from 8.21% to 33.33%, Llama-2-70B from 8.43% to 32.10%, and Mixtral-8x22B from 50% to 60.02%.Source: §5 · p. 9

Capability difficulty

  • Problem-solving shows smaller performance gaps between models, whereas self-correction remains difficult and many open-source models lack effective self-correction skills.The authors identify self-correction and planning as challenges for future model development.Source: §5 · p. 9

Capability balance

  • GPT-4-family models perform more evenly across capabilities, while weak performance in one capability tends to coincide with broader struggles.The paper describes these capabilities as highly correlated and interdependent.Source: §5 · p. 9

Model scaling

  • Increasing model size consistently improves MistralAI performance across domains, but not Llama-2 performance; Llama-2-7B surpasses larger variants on DAG-QA.The authors relate the scaling differences to the influence of training strategies and model architectures.Source: §5 · p. 9

6 Conclusion

MMAU evaluates LLM agents through 20 tasks that decompose capabilities beyond standard benchmarks. Its current scope omits interactive environments and some agent-relevant domains, and its capability decomposition still has difficulty disentangling compound capabilities.

Conclusion

  • MMAU combines application-scenario and fundamental-capability evaluation across 20 tasks to provide more granular insight into model strengths and limitations.The conclusion presents it as a test bed for reliable, thorough studies.Source: §6 · p. 9

Limitations and future work

  • MMAU does not cover all relevant domains, including interactive environments, and its capability decomposition still struggles to disentangle compound capabilities.Future work calls for interactive tasks, additional capabilities such as retrieval and memorizing, and improved decomposition methods.Source: §6 · p. 9

Ethics and Societal Impacts

  • The authors identify bias and harmful-content risks in evaluation and report screening datasets and prompts for personally identifiable information and offensive content.They note that evaluations should not inadvertently reinforce biases or propagate harmful content.Source: §6 · p. 9

A Experiment and dataset details

The appendix describes deterministic evaluation settings and task-specific procedures for tool-use, DAG QA, and self-correction. It also points to dataset-statistics and model-inventory tables without giving their contents in these passages.

A Experiment and dataset details

  • The appendix directs readers to Table 4 for MMAU dataset statistics and Table 5 for the evaluated-model list, without stating their values here.Open-source models run through VLLM on up to eight NVIDIA A100 or H100 GPUs, using Hugging Face checkpoints; evaluations use temperature 0 and greedy search.Source: A Experiment and dataset details · p. 14, Table 4 · p. 14, Table 5 · p. 14

A.1 Tool-use

  • Tool-use conversations follow a user query, candidate-function list, model response, function execution, and possible follow-up calls.Predicted and reference parameter values are normalized by removing punctuation and whitespace and lowercasing; regex rules handle some semantically equivalent open-ended answers.Source: §A.1 · p. 14

A.2 DAG QA

  • DAG QA asks the model to choose among enumerated tool plans, with reasoning and planning variants differing in their requested response format.The prompt supplies the task, possible functions, possible tool sequences, and task input; the selected plan is extracted by string matching.Source: §A.2 · p. 15

A.3 Self-correction tool-use

  • The self-correction tool-use procedure prepends an additional system message to the first user turn to encourage retries.The supplied passage introduces the retry message but does not provide its content.Source: §A.3 · p. 15
  • For non-multimodal QA models, Figure 6 reports text-question accuracy scaled by the text-question share, 207/328.The figure passage also specifies temperature 0 for code generation and QA.Source: Figure 6 · p. 15

A.4 Contest-level Coding

Contest-level coding evaluates end-to-end solutions, problem parsing, planning, solver ability, and retry-based self-correction. The Math appendix describes curation of rewritten problems, while Table 6 is identified as mapping capabilities to tasks.

A.4 Contest-level Coding

  • The contest-level dataset uses 261 valid problems from the Valid and Test splits of CodeContests, with public, private, and generated test cases.The E2E Standard task compiles and runs generated Python solutions on all test cases and reports pass@5; its model temperature is 0.3.Source: §A.4 · p. 15
  • Problem Parsing asks models to infer outputs for selected unseen test cases, then scores answer accuracy with temperature 0.The selected case is the shortest among private and generated test cases; prompts include the problem description and public examples.Source: §A.4 · p. 15
  • Planner-shift evaluates plans with GPT-4-0125-preview as the fixed solver, while Solver-shift varies solvers using a fixed GPT-4-0125-preview plan.Each setup generates one plan and five code solutions, then measures the percentage of problems solved.Source: §A.4 · p. 15
  • Retry-based self-correction returns failed solutions’ error feedback to the model when all five initial candidates fail, then evaluates pass@5.Feedback covers empty solutions, compilation errors, runtime errors, and wrong outputs.Source: §A.4 · p. 16

A.5 Math

  • The Math curation process labels seed problems by difficulty, selects simpler problems, rewrites them with more complex descriptions, and checks validity and answer consistency.The selected simple set contains 672 problems; GPT-4 generates rewrites that undergo a curation stage before the Comprehend+ set is finalized.Source: §A.5 · p. 16
  • Table 6 is identified as the mapping between MMAU capabilities and tasks, but the passage does not specify the individual mappings.Source: Table 6 · p. 16

B Prompt templates

The appendix supplies prompt templates for tool-use, DAG, data-science, contest-coding, and mathematics tasks, with task-specific inputs, outputs, and feedback. These examples show how the benchmark operationalizes different capabilities through distinct response workflows.

B.1 Tool-use

  • Tool-use prompts are generated from tool definitions, predicted calls, and tool results rather than reported as fixed concrete prompts.The appendix points to single-, parallel-, and multi-tool examples.Source: §B.1 · p. 17

B.2 DAG

  • DAG prompts distinguish concurrent tools from dependency-ordered execution and ask the model to select a plan from the available options.For example, “[A B]” denotes concurrent tools, while “[A B] -> C” makes C depend on both outputs.Source: §B.2 · pp. 17–18

B.3 Data Science and Machine Learning

  • Data-science tasks ask models to add notebook code, receive its execution results, and then answer multiple-choice questions based on those results.When execution fails, a separate feedback prompt asks the model to check the error and retry.Source: §B.3 · pp. 18, 20
  • Data-science code-generation prompts require nbformat cells and restrict allowed math and visualization packages.The response also follows a specified cell structure and multiple-choice answers use uppercase single-letter strings.Source: §B.3 · p. 19

B.4 Contest-Level Coding

  • Contest-coding problem parsing tests whether a model can infer an answer from a problem and example input, while the standard prompt requests concise thoughts and an answer.The parsing task focuses on comprehension without requiring code generation.Source: §B.4 · p. 21
  • Contest planner-shift and solver-shift prompts separate planning from code generation by supplying a plan to the solver or asking for an executable plan.Solver prompts require robust Python code that handles valid inputs beyond the examples.Source: §B.4 · pp. 21–22

B.5 Mathematics

  • Math-generation prompts ask for concise thoughts followed by an answer, with no extra explanation under the answer section.The response format separates the reasoning-oriented thoughts from the final answer.Source: §B.5 · pp. 22–23
  • Math planning prompts request a structured, executable sequence of steps, while evaluation prompts ask models to judge whether an answer matches the ground truth.The evaluation prompt allows mathematically equivalent expressions to count as matching.Source: §B.5 · p. 23

C More evaluation results

The section describes the domain-centric scoring setup and reports tool-use evaluation constraints. Some models could not be tested on all tool-use datasets under their documented prompts.

C More evaluation results

  • Domain-centric results use end-to-end standard tasks for Contest-level coding, DS&ML, Math, and DAG-QA, and a weighted average of three tool-use task types.The tool-use average combines single-, parallel-, and multi-turn multi-tool-use tasks.Source: C More evaluation results · p. 24
  • Command R and Command R+ could not be evaluated on double and triple tool-use datasets because their documented prompts do not support multi-step tool calls.The evaluation used official repository prompts for Command R, Command R+, and Hermes-2-Pro-Mistral-7B.Source: C More evaluation results · p. 24
  • Claude, Gemini, and GPT models were evaluated through their dedicated tool-use APIs.Source: C More evaluation results · p. 24

Detailed Reulst on Data Science and Machine Learning

GPT-4o achieves the highest reported accuracy across four DS&ML settings, while the section also describes contest-coding performance and the capability distinctions its task designs expose.

Detailed Reulst on Data Science and Machine Learning

  • GPT-4o leads all four DS&ML settings: E2E 66.52%, Execution Only + GPT-4 QA 64.27%, QA from Oracle Execution 88.60%, and E2E + Online Retry 70.79%.Table 9 presents the detailed DS&ML evaluation results.Source: Detailed Reulst on Data Science and… · p. 24, Table 9 · p. 25
  • Contest-level E2E scores are relatively low across models, while GPT-4o and GPT-4-Turbo perform much better; GPT-4o leads task types, especially Problem Parsing and Retry.Table 10 presents detailed contest-coding results.Source: Table 10 · p. 25, Detailed Reulst on Data Science and… · pp. 25–26
  • Mixtral-8x22B-v0.1 has the best overall performance among evaluated open-source DS&ML models; Gemini-1.5-pro and Claude3 Opus score 83.23% and 83.48% on oracle-execution QA.Source: Detailed Reulst on Data Science and… · p. 25
  • Problem Parsing and PlannerShift omit code generation, exposing understanding, reasoning, and planning; some open-source models show these capabilities despite weak E2E results.Most models improve with a separate strong solver or planner, whereas GPT-4o and GPT-4-Turbo score lower on the shifted tasks than on E2E Standard.Source: Detailed Reulst on Data Science and… · p. 26

D Data examples

The appendix presents illustrative data examples spanning tool use, mathematics, DAG QA, and contest-level problem parsing, including retry and self-correction cases.

D Data examples

Loading 2407.18961v3…