Source-linked AI summary
GAIA: a benchmark for General AI Assistants
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, Thomas Scialom
TL;DR
Existing evaluations do not adequately test whether AI assistants can robustly execute conceptually simple tasks in varied real-world settings. GAIA introduces 466 curated questions requiring reasoning, multimodal understanding, and tool use, with short factual answers. Humans achieve 92% success while GPT-4 with plugins remains below 30% on the easiest tasks and reaches 0% on the hardest.
Problem
Current benchmarks can saturate quickly, rely on closed environments or contaminated datasets, and make general-assistant evaluation difficult to interpret and scale.
Method
GAIA provides 466 human-designed, real-world questions with unique short answers, sometimes attached files, and coverage of reasoning, multimodality, coding, and tool use.
Results
92% human success contrasts with GPT-4 plus tools achieving below 30% on the easiest tasks and 0% on the hardest.
Takeaways & Limitations
GAIA clearly ranks capable assistants while leaving substantial room for improvement in general assistant performance.
Takeaways & Limitations
All questions use standard English and many rely on English web pages, so GAIA does not validate assistants for non-English speakers or the non-English web.
Abstract
from arXiv · showhide
We introduce GAIA, a benchmark for General AI Assistants that, if solved, would represent a milestone in AI research. GAIA proposes real-world questions that require a set of fundamental abilities such as reasoning, multi-modality handling, web browsing, and generally tool-use proficiency. GAIA questions are conceptually simple for humans yet challenging for most advanced AIs: we show that human respondents obtain 92\% vs. 15\% for GPT-4 equipped with plugins. This notable performance disparity contrasts with the recent trend of LLMs outperforming humans on tasks requiring professional skills in e.g. law or chemistry. GAIA's philosophy departs from the current trend in AI benchmarks suggesting to target tasks that are ever more difficult for humans. We posit that the advent of Artificial General Intelligence (AGI) hinges on a system's capability to exhibit similar robustness as the average human does on such questions. Using GAIA's methodology, we devise 466 questions and their answer. We release our questions while retaining answers to 300 of them to power a leader-board available at https://huggingface.co/gaia-benchmark.
1 Introduction
GAIA targets real-world assistant questions that are conceptually simple for humans but require robust reasoning, multimodal handling, and tool use. Its curated, factual, and difficult-to-game design exposes a large gap between human and LLM performance.
- GAIA addresses evaluation challenges arising as LLMs rapidly break existing benchmarks and increasingly use tools such as web browsers and code interpreters.
- 466 carefully crafted questions have unique factual answers, making them challenging for AI systems while supporting simple and robust automatic evaluation.
- GAIA uses real-world questions requiring browsing, multimodal processing, or multistep reasoning rather than restricting evaluation to closed, synthetic environments.
- GAIA emphasizes interpretability through few highly curated questions, associated reasoning traces, and conceptually simple tasks that non-expert annotators nearly solve perfectly.
- GAIA seeks to limit gameability because diverse multistep tasks, required accuracy, reasoning traces, and absence from plain-text internet sources hinder brute force and contamination.
- Factoid, concise, and unambiguous answers enable fast factual evaluation and zero-shot testing with less dependence on prompts or benchmark implementation.
- 92% human success contrasts with GPT-4 plus tools achieving below 30% on the easiest tasks and 0% on the hardest.
2 Related work
Prior evaluations increasingly saturate, risk contamination, or depend on costly and potentially unreliable human or model-based judgments. Existing general-assistant benchmarks also often use closed environments or repurposed datasets, limiting assessment of real-world generality.
- Static language benchmarks have been saturated at human level increasingly quickly as LLM capabilities progress.
- Benchmark results are difficult to aggregate meaningfully and vulnerable to data leakage, while human evaluation is time-consuming and difficult to scale.
- Model-based evaluation can scale beyond human judgment but depends on a more capable evaluator LLM whose shortcomings may produce subtly incorrect results.
- General-assistant evaluations often rely on closed systems, specific API calls, fixed answer procedures, or existing datasets that risk training contamination.
- Closed-box environments can measure learned API usage rather than general results grounded in real-world interactions.
3 GAIA
GAIA is a curated benchmark for general AI assistants, designed around conceptually simple real-world questions that require robust reasoning, multimodal understanding, and tool use. Its questions are intended to be interpretable, difficult to game, easy to evaluate, and organized by increasing solution complexity.
- Benchmark overview: GAIA comprises 466 human-designed questions with short, single correct answers, sometimes accompanied by files such as images or spreadsheets.The questions cover use cases including daily personal tasks, science, and general knowledge.
- Design principles: Questions target conceptually simple but varied real-world tasks requiring information gathering and transformation across documents or the changing web.The design emphasizes adaptation through reasoning, multimodality understanding, and diverse tool use rather than specialized skills.
- Design principles: GAIA seeks interpretability and resistance to memorization by requiring successful multistep actions whose concise answers are absent from pretraining text and easy to verify.Its evaluation uses quasi-exact matching against a single ground-truth answer, with normalization determined by the answer type.
- Capabilities coverage: The benchmark covers advanced reasoning, multimodal processing, coding, web browsing, and varied data modalities, while excluding website actions beyond clicking.Different capability combinations may solve the same question, so the benchmark does not prescribe a fixed capability list for each item.
- Increasing difficulty: Difficulty increases across three levels according to annotator-estimated solution steps and tools, from at most one tool and five steps at Level 1 to arbitrarily long action sequences at Level 3.These definitions are proxies rather than hard constraints, since complex navigation can raise a question’s assigned level.
- Building and validating questions: Question quality is validated by independent annotators checking agreement on a single answer, with 68% accepted as-is and remaining questions corrected or removed.Web-based questions may require versioned evidence because sources can change over time, and access restrictions for bots should be respected.
4 LLMs results on GAIA
GAIA evaluates LLM assistants across real-world, multi-step questions using varied capabilities and tools. Current models perform poorly overall, while tool augmentation improves accuracy but remains difficult to reproduce and insufficient relative to humans.
- Evaluation setup: GAIA evaluates assistants through zero-shot prompting, comparing GPT4 with and without plugins, AutoGPT4, human annotators, and web search.When an API was available, the authors ran models three times and reported average results; GPT4 plugins required manual queries.
- Evaluation setup: The GPT4-with-plugins score is an oracle estimate because plugins were manually selected for each question and were unstable over time.The authors therefore describe it as an estimate of potential with stable, automatically selected plugins rather than a reproducible result.
- Overall results: GAIA difficulty levels correlate with current-model performance, while humans excel across all levels and the benchmark leaves substantial room for improvement.The results are presented in Figure 4 and further detailed in Appendix Table 4.
- Overall results: Web search can yield Level 1 answers, but fails on slightly more complex queries and is slower than a typical LLM assistant.Users must skim the first search results to derive answers.
- Tool use: Tool and web augmentation improves GPT4 answer accuracy and enables behaviors such as backtracking, query refinement, and longer plan execution.The remaining gap with humans indicates that this potential is not yet fully realized.
- Tool use: AutoGPT4 performs disappointingly at Levels 1 and 2 relative to GPT4 without plugins, is slower, and is less efficient than human collaboration with GPT4 plugins.The authors attribute the discrepancy possibly to AutoGPT4's API, prompt, and generation parameters, and call for further evaluation.
- Capability analysis: GPT4 cannot handle files and multimodality, but solves some web-browsing questions by memorizing information needed to combine intermediate steps.Figure 5 reports scores broken down by capability.
5 Discussion
The discussion examines GAIA's implications for reproducible, evolving, and unified evaluation of AI systems. It emphasizes that exact-answer tasks test full automation while exposing limitations of closed assistants and static benchmarks.
- Reproducibility for closed-source assistants: Closed-source assistants and changing plugins make evaluations difficult to reproduce, although GAIA's single-answer scoring is robust to token-generation randomness.The authors note that model capabilities and plugin availability can change over time.
- Static versus dynamic benchmarks: GAIA's curated scale is smaller than MMLU's, but its open questions require a single correct answer and are presented as harder than multiple-choice evaluation.The discussion contrasts GAIA's hundreds of questions with MMLU's nearly 15,000.
- Static versus dynamic benchmarks: The authors argue that static benchmarks become obsolete and propose evolving GAIA by removing broken questions and adding new ones year by year.This is intended to better assess generalization and robustness.
- Towards unified evaluation of generative models: GAIA evaluates complete AI systems rather than attributing errors to individual modules, anticipating future integration of language, vision, and other capabilities.The discussion frames automatic, factual, interpretable evaluation of complex generations as a broader challenge.
- Partial versus full automation: Because GAIA accepts no approximation in its answers, solving it requires full automation rather than partial automation with humans remaining in the loop.The discussion connects this distinction to the broader socioeconomic implications of automating human activities.
6 Limitations
GAIA identifies remaining evaluation limitations involving answer traces, question ambiguity, and linguistic and cultural coverage.
- GAIA does not evaluate the trace leading to an answer because different paths can produce the same correct answer and are difficult to grade simply.The authors leave human- and model-based plan evaluations for future work.
- Designing unambiguous questions requires multiple annotation rounds, yet possible ambiguities remain despite this process.The authors describe the annotation cost as fixed and probably small compared with repeated untrustworthy evaluations.
- Under-specified user questions and source-selection judgments remain difficult to evaluate factually and are left for future work.
- GAIA uses only standard English and often English web pages, so it does not validate assistants for non-English speakers, non-English web content, or English dialectal variation.The authors characterize GAIA as a first step rather than an absolute proof of assistant success.
A Extended related work
Related work has pursued general-purpose assistants through stronger single agents, multi-agent collaboration, and tool-augmented language models.
- Single-agent approaches improve LLM capabilities through Chain of Thought prompting or equivalent mechanisms.
- Multi-agent approaches use debate and collaboration to reach better conclusions for user queries.
- Tool-augmented assistants extend LLMs with specialized tools, planning components, multimodal models, web search, or collections of tools and APIs.
B Datacard
The GAIA datacard centralizes dataset information and documents its curation context, language variety, annotator population, and text-characteristics references.
- The datacard follows Bender and Friedman’s creation framework to summarize and centralize information relevant to dataset analysis.
- GAIA’s curation rationale is detailed in Section 3.4 and Appendix D.
- Annotators were based in the United States, and questions, answers, and metadata were written in mainstream English, likely en-US.The authors are French, which might have contributed non-standard English phrasing in questions or answers.
- Building GAIA involved Curators who devised questions and answers and Annotators who independently assessed question non-ambiguity.
- The reported Curators and Annotators population was 57% male and 43% female.
- Information about text characteristics is provided in Appendix C.
C Extended description of GAIA
GAIA characterizes required capabilities, associated tools, file inputs, and relationships between answering effort, tool use, and time.
- Annotators’ steps and tools were used to define GAIA capabilities and provide corresponding tool examples.
- Capabilities: Web browsing includes searching and browsing websites, with examples including web browsers, search engines, YouTube access, and Google Street View.
- Capabilities: Multi-modality covers non-text data understanding, including speech-to-text, video recognition, image recognition, OCR, and Google Street View.
- Capabilities: Coding covers code execution through tools such as Python, calculators, compilers, encoders, and scripts.
- Capabilities: Diverse filetype reading covers user-provided or web-found PDFs, spreadsheets, PowerPoint files, CSVs, and text files.
- Capabilities: Capability categories are indicative rather than a perfect typology because one tool can belong to multiple categories.Google Street View is given as an example involving both web browsing and multi-modality.
- Filetypes: Questions with additional files are summarized through their file-type distribution in Figure 6.
- Difficulty of the questions: Answering time correlates with the number of steps taken, while its relationship with the number of different tools used is less clear.
D Extended description of our question design framework
GAIA questions are designed for factual, robust, zero-shot evaluation while remaining challenging for AI systems. The framework uses human creation, independent validation, and repair or removal of ambiguous questions.
- Question creation criteria: Annotators are instructed to ground questions in sources of truth and avoid answers already available online verbatim.For harder questions, the framework permits combining multiple sources of truth.
- Question creation criteria: Questions target concise, factual, time-stable, unambiguous answers that support robust evaluation.Design instructions require answers to be numbers or at most a few words and prohibit answers that change over time.
- Question creation criteria: Questions should be interesting and answerable by a human annotator in a reasonable amount of time.
- Validation phase: Two independent annotators validate each question, with disagreements triggering repair when possible or removal otherwise.
- Annotation effort: Creating one question, including validation by two supplementary annotators and potential repairs, requires an estimated two hours of annotator time.
- Validation phase: 623 newly crafted questions were validated through 1,246 annotations in the validation phase.
D.1 Extended evaluation
The evaluation section reports baseline scores and illustrates assistant behavior on GAIA questions. Examples include web-search refinement, GPT-4's difficulty with puzzles, and differences in output interpretability between GPT-4 and AutoGPT.
- Output examples: AutoGPT outputs are longer, denser, and less interpretable than GPT4 outputs in the provided examples.AutoGPT examples are supplied in supplementary material for the same GAIA question as Figure 9.
- Web browsing examples: Proper web search is described as very effective for answering GAIA questions.The associated reasoning trace was obtained with the current GPT4 web browsing version.
- Web browsing examples: An earlier GPT4 browsing-plugin version could refine its initial query using information it found, but that trace could not be reproduced with the new version.
- Puzzle examples: GPT4 and other assistants struggle on puzzles, which are often Level 1 questions.