GAIA: a benchmark for General AI Assistants

1 Introduction

• Existing benchmarks are rapidly surpassed, while increasing human difficulty does not reliably challenge LLMs and makes open-ended evaluation harder.Human evaluation becomes less feasible as output length and required skills increase. Model-based evaluation depends on stronger models and can exhibit presentation-order bias.Source: §1 · p. 1
• GAIA targets conceptually simple tasks requiring accurate execution of complex action sequences, including browsing the changing web, handling multimodality, and multistep reasoning.These practical assistant tasks involve large combinatorial spaces while producing outputs that are easy to validate.Source: §1 · p. 2
• 466 carefully crafted questions and a design methodology define GAIA, with unique factual answers enabling simple, robust automatic evaluation.The release includes 166 annotated developer questions and 300 questions without annotations for a leaderboard. Concise, unambiguous factoid answers enable fast factual evaluation, while zero-shot answering is intended to limit setup sensitivity.Source: §1 · pp. 2–3
• GAIA supports interpretability through conceptually simple tasks, associated reasoning traces, and a small, highly curated question set on which nonexpert annotators score nearly perfectly.Source: §1 · p. 2
• GAIA targets non-gameability through diverse multistep tasks, checkable reasoning traces, precise answers, and answers absent verbatim from the internet.Task diversity makes brute forcing difficult; reasoning checks and answer requirements are intended to prevent contamination from yielding correct answers without successful execution.Source: §1 · p. 2
• 92% average human success contrasts with tool-equipped GPT4 achieving at most 30% on GAIA’s easiest tasks and 0% on its hardest.Humans typically take 6 minutes for the simplest questions and 17 minutes for the most complex. The authors argue that solving GAIA would indicate competent General AI under Morris et al. (2023).Source: §1 · p. 3

Evaluating Large Language Models

• Static language-model benchmarks saturate increasingly quickly, and harder evaluations requiring professional knowledge have also seen models surpass human performance.Models surpassed humans on GLUE within a year. Professional-level evaluations include MMLU (Hendrycks et al., 2021), while reports describe plausible US bar-exam passage and scores exceeding the USMLE passing threshold.Source: §2 · p. 3
• Broader evaluation faces difficult aggregation and data leakage, costly human assessment, and model-based judgments that inherit evaluator weaknesses.Model-based evaluation requires a more capable evaluator, often GPT4, whose shortcomings can produce subtly incorrect results.Source: §2 · p. 3

Evaluating General Assistants

• Assistant benchmarks often reuse datasets or prescribe APIs, risking contamination and failing to ensure that tool use is actually tested.ToolQA and Gentopia combine existing datasets with human annotations. Gorilla evaluates specific API calls through APIBench, while API-Bank supplies an API pool.Source: §2 · p. 3
• GAIA leaves possible APIs unspecified and relies on real-world interactions, unlike closed-environment evaluations that risk measuring familiarity with particular APIs.This distinction targets general results grounded in real-world interactions rather than proficiency within a prescribed environment.Source: §2 · p. 4
• OpenAGI is closer to GAIA through multistep, multimodal tasks, but focuses on current model capabilities rather than upcoming advancements.OpenAGI provides both a platform and a benchmark spanning multiple modalities and capabilities.Source: §2 · p. 4

3.1 A convenient yet challenging benchmark for general AI assistants

• 466 human-designed questions test realistic assistant tasks using zero-shot prompts, sometimes accompanied by files, and require short, uniquely correct answers.Questions prioritize conceptually simple but potentially tedious information gathering and transformation over specialized skills. A small, curated question set and understandable reasoning traces support interpretability.Source: §3.1 · p. 4
• GAIA targets memorization resistance through multistep tasks whose answers are absent by design from current pretraining data.Task diversity, precise answers, and inspectable reasoning traces mitigate—but do not eliminate—contamination risk. The question-writing guidelines enable replacement questions if substantial memorization occurs.Source: §3.1 · p. 4

3.2 Evaluation

• Evaluation uses quasi-exact matching with answer-type-specific normalization, enabling automated, fast, factual scoring.Answers are short strings, numbers, or comma-separated lists unless otherwise specified; a system prompt communicates formatting requirements.Source: §3.2 · p. 5

3.3 Composition of GAIA

• GAIA requires reasoning, multimodal understanding, coding, and tool use, but excludes website actions beyond clicks and does not benchmark tool usage finely.Evidence includes PDFs, spreadsheets, images, videos, and audio; different capability combinations can solve the same question. The English-only dataset seeks diverse domains and cultures and includes tasks potentially useful to physically impaired people.Source: §3.3 · pp. 6–7
• Three loosely defined difficulty levels use annotators’ step and tool counts as proxies, acknowledging that questions admit different solution paths.Level 1 generally needs no tools, or at most one tool and five steps; Level 2 typically combines tools across roughly five to ten steps. Level 3 permits arbitrarily long action sequences and any number of tools.Source: §3.3 · p. 6

3.4 Building and extending GAIA

• Human question creators use varied sources of truth and annotate answers, tools, steps, and completion time.Authors supplied initial examples and instructions to annotators. Sources include durable web pages, attached documents, and self-contained puzzles; avoiding a fixed source list encourages diversity and discourages memorization.Source: §3.4 · p. 7
• 68% of questions passed validation unchanged; validation required agreement between the creator and two additional annotators answering independently.Other questions were corrected or removed, and estimated annotator success across difficulty levels was 92%, reported as GAIA’s human score. Creation, validation, and repairs required an estimated two hours per question.Source: §3.4 · p. 7
• Web-based questions favor durable evidence, specify versions when needed, and check that evidence access is not restricted by robots.txt.Changing pages can remove required evidence, while website owners may request that bots avoid parts or all of a site.Source: §3.4 · p. 7

Evaluation setup

• Evaluation compares GPT4, with and without plugins, and GPT4-backed AutoGPT against human annotators and web search.A prefix specifies answer format, and API-accessible models are run three times with results averaged. Plugin-enabled GPT4 requires manual ChatGPT queries and task-specific selection between Advanced Data Analysis and at most three third-party plugins; changing plugin availability makes its score an oracle estimate rather than an easily reproducible result.Source: §4 · p. 8

Results

• Humans excel across all GAIA difficulty levels, while current LLMs perform poorly, leaving substantial room for improvement.Difficulty levels, loosely defined by the number of steps and capabilities required, correlate with current models’ performance, supporting their validity.Source: Figure 4 · p. 8, §4 · p. 8
• Human web search can yield answers for Level 1 but fails on slightly more complex queries and is slightly slower than a typical LLM assistant.The baseline checks whether answers can be deduced from the first results page; users must spend time skimming those results.Source: §4 · pp. 8–9
• Augmenting GPT4 with tools or web access improves answer accuracy and enables backtracking, query refinement, and relatively long plan execution.The remaining gap with humans indicates that these capabilities are not yet fully realized.Source: §4 · p. 9
• GPT4 solves some questions requiring annotator web browsing through memorized information, despite limitations in file handling and multimodality.These answers require combining memorized information. Nonzero file-reading and multimodality scores for models without tools reflect alternative solutions that differ from annotators’ approaches.Source: Figure 5 · p. 9, §4 · p. 9
• AutoGPT4’s automatic tool use underperformed GPT4 without plugins at Levels 1 and 2, while also running slower than other LLMs.The authors suggest API prompting and generation parameters as possible explanations requiring reevaluation. Human collaboration with GPT4 plus plugins appeared to offer the best score–time trade-off.Source: §4 · p. 9

Reproducibility for closed-source assistants

• Changing closed-source assistants threaten reproducibility, while GAIA limits token-generation randomness by scoring only a uniquely correct final answer.Regularly changing plugins and their lack of API access further complicate reproduction. Real-world-dependent benchmarks that decay over time could make reproducibility still more elusive.Source: §5 · p. 9

Static versus dynamic benchmarks

• GAIA may decay through training-data contamination or disappearing web information, motivating yearly removal of broken questions and addition of new ones.Its carefully curated, single-answer open questions prioritize quality over quantity. Updating the benchmark could better assess generalization and robustness, with community contributions helping expand its question set.Source: §5 · pp. 9–10

Towards unified evaluation of generative models

• GAIA evaluates complete AI systems rather than particular architectures, accommodating both external tools and increasingly integrated capabilities.Whole-system evaluation does not attribute errors to individual browsing or vision modules. The authors propose evaluating image generators through sequences of modifications followed by an unambiguous natural-language question answerable only when those modifications succeed.Source: §5 · p. 10

Partial versus full automation

• Solving GAIA requires full automation rather than human-in-the-loop partial automation, because answers allow no approximation.Unpredictable failures still impede full automation despite strong neural-system performance. Further automation will reshape the socioeconomic landscape; the risk that technology owners capture gains rather than workers motivates open source.Source: §5 · p. 10

Missing evaluations

• GAIA evaluates final answers rather than solution traces because multiple valid paths make grading difficult, prioritizing ease of use over process evaluation.Human and model-based evaluation of plans remains future work. Limited expert-knowledge requirements and access to ground-truth answers could simplify judging.Source: §6 · p. 10
• Evaluation covers only the strongest available tool-enabled LLMs, while missing detailed OpenAI API tool-call logs prevent fine-grained analysis.The authors seek to include additional models with sufficient tool-use capabilities and logging, especially open-source models.Source: §6 · p. 10

On the cost of designing unambiguous questions

• Two annotation rounds aim to remove ambiguity: one annotator designs each question, then two additional annotators independently answer and disambiguate it, yet ambiguities remain.The authors consider this fixed cost probably small relative to repeated untrustworthy evaluations and human annotation essential for diverse, grounded questions. Synthetic training data could relax the unambiguity requirement.Source: §6 · p. 10
• Details needed to guarantee unique answers can make GAIA questions unnatural, unlike underspecified user requests requiring source citation or selection of trustworthy sources.Factually evaluating these more realistic responses remains difficult and is left for future work.Source: §6 · pp. 10–11

Lack of linguistic and cultural diversity

• 80% of the global population—non-English speakers—are outside GAIA’s validated scope because its questions use only standard English and often English webpages.GAIA also cannot validate usefulness on non-English web content, approximately half of the web, or English dialects. It should not be considered absolute proof of assistants’ success.Source: §6 · p. 11

Large Language Models as General Assistants

• Single-agent approaches improve LLM capabilities through Chain of Thought prompting or equivalent mechanisms.Examples include GPT-Engineer and AutoGPT.Source: A Extended related work · p. 17
• Multi-agent approaches use several LLM agents to debate and jointly reach better conclusions when answering user queries.The cited work includes Hong et al., 2023; Chan et al., 2023; Talebirad and Nadiri, 2023.Source: A Extended related work · p. 17
• Tool-augmented assistants add planning, multimodal capabilities, web search, and API access, while dedicated libraries provide tooling infrastructure.Toolformer (Schick et al., 2023) is fine-tuned for tool use, ViperGPT (Sur´ıs et al., 2023) generates API calls through code, and HuggingGPT invokes other ML models. Tooling libraries include OpenAI plugins, SemanticKernel, Langchain, and MiniChain.Source: A Extended related work · p. 17

Datacard framework

• The datacard follows to summarize and centralize information relevant to analyzing the dataset.The curation rationale is referenced in Section 3.4 and Appendix D.Source: B Datacard · p. 17

Language variety

• All questions, answers, and metadata use mainstream English, likely en-US, with possible non-standard phrasing from French authors who are not native English speakers.Annotators were all US-based, but their nationalities were not provided.Source: B Datacard · p. 17

Curators and Annotators demographic

• Curators devised questions and answers, while annotators independently checked questions for non-ambiguity, following the role definitions in.57% Male and 43% Female was the reported gender distribution for the contributor population.Source: B Datacard · p. 17

Description of capabilities

• Annotators’ reported tools define GAIA’s capability categories: web browsing, multi-modality, coding, diverse filetype reading, and tasks achievable by non-augmented LLMs.Annotators recorded both the steps followed and the tools used when answering questions.Source: C Extended description of GAIA · p. 18

Filetypes

• Some GAIA questions include supplementary files; file-reading capabilities cover formats such as PDF, Excel, PowerPoint, CSV, and text.Figure 6 presents the initial distribution of file types.Source: C Extended description of GAIA · p. 18, Figure 6 · p. 18

Description of capabilities

• Capability categories overlap rather than forming a perfect question typology: Google Street View requires both web browsing and multi-modality.The categories indicate capabilities required by GAIA rather than mutually exclusive question classes.Source: C Extended description of GAIA · p. 18

Difficulty of the questions

• Answering time correlates with the number of solution steps, but its relationship with the number of different tools is less clear.Using multiple tools does not necessarily require more time.Source: C Extended description of GAIA · p. 18, Figure 8 · p. 19, Figure 7 · p. 19

Question creation phase

• Questions must expand beyond existing variations, rely on sources of truth, and have answers unavailable online in plain text.Annotators received seed questions, answered their own creations, and were encouraged to combine sources for Levels 2 and 3. A later instruction required checking robots.txt for AI-assistant accessibility.Source: D Extended description of our question… · p. 19
• Answers must be unambiguous, time-stable numbers or at most a few words to support robust evaluation.The stability requirement includes potential deletion of the source of truth.Source: D Extended description of our question… · p. 19
• Questions should demonstrate useful AI-assistant capabilities while remaining answerable by a human annotator within a reasonable time.“Interesting” questions are defined as those whose successful completion by an assistant would substantially help the reader.Source: D Extended description of our question… · p. 19

Validation phase

• Two new independent annotators validate each question; disagreement with the original answer, unless caused by human error, requires repair or removal.Validation covered 623 newly crafted questions and 1246 annotations. Creation, validation, and potential repairs were estimated to require two hours of annotator time per question.Source: D Extended description of our question… · p. 19, Table 3 · p. 21

Baseline reporting

• Baseline reporting averages three runs when direct API access exists; human scores measure validation annotators’ correct-answer fraction on valid questions.GPT4 + plugins used manually selected plugins and is not exactly reproducible. API timings average 20 questions at one time point, comparing GPT4 with other method types rather than GPT4 Turbo speed.Source: Table 4 · p. 21

• Proper web search is very effective for answering GAIA questions, as illustrated by a reasoning trace from the current GPT4 web-browsing version.Source: Figure 9 · p. 22
• GPT4’s former official browsing plugin could refine its initial query using retrieved information, but its illustrated trace was not reproducible with the new version.Source: Figure 10 · p. 23
• GPT4 and other assistants struggle with puzzles, which are often Level 1 questions.Source: Figure 11 · p. 24