Source-linked AI summary
OAgents: An Empirical Study of Building Effective Agents
He Zhu, Tianrui Qin, King Zhu, Heyuan Huang, Yeyi Guan, Jinxiang Xia, Yi Yao, Hanhao Li, Ningning Wang, Pai Liu, Tianhao Peng, Xin Gui, Xiaowan Li, Yuhui Liu, Yuchen Eleanor Jiang, Jun Wang, Changwang Zhang, Xiangru Tang, Ge Zhang, Jian Yang, Minghao Liu, Xitong Gao, Jiaheng Liu, Wangchunshu Zhou
TL;DR
Agent research lacks standardized designs and evaluation practices, making comparisons and reproduction difficult. The paper studies agent designs and evaluation on GAIA and BrowseComp and introduces a robust protocol and OAgents. OAgents achieves state-of-the-art performance among open-source frameworks on both benchmarks.
Problem
Agent research lacks standardized designs and implementation and evaluation details, making it difficult to attribute performance differences and reproduce results.
Source: §1 · p. 1Method
The paper systematically studies agent components and evaluation practices on GAIA and BrowseComp, proposes a robust evaluation protocol, and releases the modular OAgents framework.
Source: §1 · p. 3, §6 · p. 13Results
OAgents achieves state-of-the-art performance among open-source frameworks on GAIA and BrowseComp; on GAIA, OAgents-Pass@3 scores 73.93% overall average.
Source: §1 · p. 2, §4.2 · p. 8Takeaways & Limitations
OAgents is intended to facilitate apples-to-apples comparisons and standardize evaluation protocols for future agent research.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Recently, Agentic AI has become an increasingly popular research field. However, we argue that current agent research practices lack standardization and scientific rigor, making it hard to conduct fair comparisons among methods. As a result, it is still unclear how different design choices in agent frameworks affect effectiveness, and measuring their progress remains challenging. In this work, we conduct a systematic empirical study on GAIA benchmark and BrowseComp to examine the impact of popular design choices in key agent components in a fair and rigorous manner. We find that the lack of a standard evaluation protocol makes previous works, even open-sourced ones, non-reproducible, with significant variance between random runs. Therefore, we introduce a more robust evaluation protocol to stabilize comparisons. Our study reveals which components and designs are crucial for effective agents, while others are redundant, despite seeming logical. Based on our findings, we build and open-source OAgents, a new foundation agent framework that achieves state-of-the-art performance among open-source projects. OAgents offers a modular design for various agent components, promoting future research in Agentic AI.
1 Introduction
Agent research lacks standardized designs and implementation and evaluation details, hindering attribution and reproducibility. The paper compares agent designs and evaluation practices, then introduces OAgents as a modular framework informed by its study.
1 Introduction
- The study compares key agent-component designs, investigates LLM backbones and their combinations, and analyzes evaluation practices and experimental stability.It also offers principles for decomposing, analyzing, and optimizing agent designs.Source: §1 · pp. 2–3
- OAgents achieves state-of-the-art performance among open-source frameworks on GAIA and BrowseComp, ranking first on GAIA.The reported GAIA result is OAgents-Pass@3, with an overall average score of 73.93%.Source: §1 · pp. 2–3
- OAgents modularizes planning, tools, memory, and test-time scaling, with plan revision, task decomposition, browsing, document parsing, and adaptive memory among its features.The framework also includes static and dynamic planning, search strategies, and reflection or self-refinement mechanisms.Source: §1 · p. 2, Figure 1 · p. 2
- Inference-parameter optimization and majority voting are proposed to reduce experimental variance and make agent evaluations more reliable and consistent.The paper identifies inconsistent run counts, failure handling, and result aggregation as evaluation choices that can substantially affect reported performance.Source: §1 · pp. 2–3
2 Related Work
Related work covers agent pipelines, multi-agent systems, and frameworks evaluated on GAIA. These approaches vary in how they organize planning, memory, tools, and collaboration.
2 Related Work
- Agent-pipeline work combines reasoning and action [35], task decomposition and tool invocation, memory and reflection [21] [12], or search over reasoning and action steps [37].The approaches described include long- and short-term memory, reflection from execution observations, and Monte Carlo Tree Search.Source: §2 · p. 3
- Multi-agent systems use leader-worker teams [6], dynamically adapted teams [13], iterative collaboration stages [4], or structured outputs to limit unproductive chatter [8].The cited work reports benefits for leader-based teams, contributor ranking, and structured collaboration.Source: §2 · p. 3
- GAIA frameworks span closed-source and open-source systems [5] [24] [30], with varied modular, multi-agent, and task-reallocation designs.Examples include Smolagents, Magentic-One [5], Trase-Agent, TapeAgents, AutoAgent [24], h2oGPTe-Agent, and Owl.Source: §2 · p. 3
- Alita [18] reports strong results using a self-evolving MCP Box, but its algorithm and implementation details are undisclosed.The authors describe that nondisclosure as inconsistent with this paper's motivation.Source: §2 · p. 4
3 Building Effective Agents
The section frames agent design along factual acquisition capacity (FAC) and logical reasoning fidelity (LRF), then details tool, planning, memory, and test-time-scaling designs. Table 1 is identified as a GAIA performance comparison.
3.1 Factual Acquisition Capacity (FAC)
- FAC is shaped by tool heterogeneity and orchestration scalability, with toolset limitations imposing hard boundaries on factual knowledge acquisition.The framework identifies multimodal tool interoperability and search efficacy as operational vectors bounding knowledge acquisition.Source: §3 · p. 4, §3.1 · p. 5
3.1.1 Multimodal Toolkit
- The multimodal toolkit processes text, speech, images, and video through synchronized cross-modal semantic parsing rather than only converting non-text inputs into text.Image and video tools extract features from their respective inputs for the agent.Source: §3.1.1 · p. 5
3.1.2 Search Agent Framework
- The Search Agent framework combines multi-source retrieval, query refinement, and minimalist browsing to support web search.Its search sources include commercial APIs and archival systems; browsing uses Search, Visit, and Read functions.Source: §3.1.2 · pp. 5–6
3.2 Logical Reasoning Fidelity (LRF)
- LRF combines planning, memory, and test-time scaling to address logical consistency, environmental adaptability, and efficiency–accuracy trade-offs.The section also identifies plan and memory as elements of this triadic architecture.Source: §3 · p. 5, §3.2 · p. 6
3.2.1 Dynamic Plan Generation
- Dynamic planning decomposes tasks into dependent subtasks, revises plans using recent observations, and checks intermediate outputs against global constraints.Past execution trajectories also provide heuristic guidelines that influence action selection and are described as helping avoid known pitfalls.Source: §3.2.1 · p. 6
3.2.2 Memory-augmented Knowledge System
- The hierarchical memory system combines current memory, summarization, vectorized retrieval, and long-term memory to organize and retrieve knowledge at multiple abstraction levels.Summarization extracts high-salience knowledge, while long-term memory integrates historical insights with current memory.Source: §3.2.2 · pp. 6–7
3.2.3 Test-Time Scaling
- Test-time scaling combines diverse LLM policies, process-based step rewards, and real-time reflection to broaden solution spaces and refine reasoning trajectories.The cited passages describe these mechanisms as improving outcome quality, solution accuracy, and error detection with on-the-fly adjustments.Source: §3.2.3 · p. 7
4 Empirical Study
The empirical study evaluates OAgents on GAIA and BrowseComp and examines how agent components and evaluation choices relate to performance. It reports results for the full framework and component-level studies.
4.1 Experimental Setup
- GAIA uses exact-match evaluation; unless otherwise stated, results report average Pass@1 from a single run, with retries allowed for empty or “Unable to determine” answers.The selected baselines include open-source systems and closed-source frameworks.Source: §4.1 · pp. 7–8
4.2 Main Results
- 73.93% average: OAgents-Pass@3 achieves the highest overall GAIA score, outperforming open- and closed-source frameworks.On Level 1, OAgents reaches 83.02%, tying the best-performing frameworks; among open-source systems, it leads OWL-Roleplaying (58.18%).Source: §4.2 · p. 8
4.2.1 FAC Evaluations
- 74.07% cross-modal accuracy versus 48.15% for the baseline; audio question-answering accuracy rises from 0% to 100% (3/3).Across base models, OAgents gains 8.09% overall with GPT-4o, 9.09% average with Gemini-2.5, and 20.61% with Claude-3-7.Source: §4.2.1 · pp. 8, 10, Table 2 · p. 9
- Search studies report a 9.3% Level 2 advantage for Jina reader, a +7.69% Level 3 gain from multi-source search, and query-optimization gains of 7.55% at Level 1 and 2.31% at Level 2.The multi-source comparison includes configurations with k = 3 and k = 5 search sources.Source: §4.2.1 · pp. 8–9, Table 3 · p. 9, Table 4 · p. 9
4.2.2 LRF Evaluations
- Plan tips improve performance by 14.54%; strategic plan review improves overall accuracy by 3.64% over static workflow, and subtask decomposition adds 2.42% over baseline.Static workflow uses the same manually designed workflow for all tasks.Source: §4.2.2 · p. 10, Table 7 · p. 11
- 55.76% average accuracy with long-term memory, versus 51.52% without memory summarization and 53.33% with memory retrieval.Memory summarization alone raises average accuracy to 52.12%.Source: §4.2.2 · pp. 10–11, Figure 2 · p. 11
4.3 Evaluations on BrowseComp
- On the more challenging BrowseComp benchmark, OAgents significantly improves the model’s web-browsing ability.The search-agent evaluation reports results on a BrowseComp subset.Source: §4.3 · p. 11, Table 8 · p. 12
4.2.2 LRF Evaluations
- Best-of-N improves overall performance by 1.82% with BO2 and 5.19% with BO4, while reflection improves it by 3.03% but lowers Level 3 performance by 6.62%.Neither strategy substantially improves Level 3 performance.Source: §4.2.2 · p. 11, Figure 3 · p. 12
5 GAIA Benchmark
GAIA results are difficult to compare because frameworks report different Pass@K metrics. OAgents is evaluated at Pass@3, while OWL is reimplemented for Pass@1 comparison.
5 GAIA Benchmark
- Different Pass@K criteria, including Pass@1, Pass@3, and Pass@5, complicate fair comparisons and obscure frameworks’ actual capabilities.The section identifies inconsistent evaluation metrics as a transparency problem.Source: §5 · p. 12
- OWL was reimplemented to obtain Pass@1 performance, while OAgents was evaluated under Pass@3.Table 9 summarizes performance under different Pass@K metrics.Source: §5 · p. 12, Table 9 · p. 12
- OAgents demonstrates competitive performance among open-source frameworks under Pass@3.The comparison builds on an integrated multimodal toolkit, multi-source information retrieval, and test-time scaling strategies.Source: §5 · pp. 12–13
6 Conclusion
The paper studies agent design on GAIA and BrowseComp and proposes a robust evaluation protocol. It introduces OAgents, an open-source modular framework that achieves state-of-the-art performance on GAIA.
6 Conclusion
- The study examines GAIA and BrowseComp, identifies key agent components, and proposes a robust evaluation protocol.The conclusion names planning, memory, and tool use as key components.Source: §6 · p. 13
- OAgents is an open-source modular agent framework reported to achieve state-of-the-art performance on GAIA (73.93).Source: §6 · p. 13
- OAgents is presented as a foundation for future research in the agentic agent area.Source: §6 · p. 13
A.1 Search Agent
The appendix describes SearchAgent’s retrieval and browsing subsystems, then outlines periodic plan revision for adapting to new observations.
A.1 Search Agent
- SearchAgent combines commercial and archival retrieval sources, routing by temporal constraints and domain-specific coverage needs.The sources include commercial search APIs and historical snapshots from the Wayback Machine.Source: §A.1 · p. 18
- Query refinement uses prompt-based reflection to resolve ambiguity and expansion to generate morphological, semantic, and domain-specific variants.The browsing interface also consolidates functionality into Search, Visit, and Read while removing non-essential operations.Source: §A.1 · p. 18
A.2 Strategic Plan Review
- The plan is periodically revised every N steps using recent observations, updating the sequence of subtasks during execution.Execution interleaves actions and reasoning under a ReAct framework.Source: §A.2 · p. 18
A.3 Subtask Decomposition.
Subtask decomposition represents a goal as interdependent subtasks, schedules executable subsets, and checks intermediate results against global constraints.
A.3 Subtask Decomposition.
- The goal is decomposed into interdependent subtasks represented by a dependency graph whose edges encode precedence constraints.Source: §A.3 · p. 19
- At each reasoning step, non-conflicting subtasks can be scheduled when their dependencies are satisfied, with intermediate outputs refined against global constraints.Completed-subtask results are represented as structured knowledge and checked for consistency with the overarching goal.Source: §A.3 · p. 19
- These mechanisms strengthen planning for long-horizon reasoning and support more effective, resilient decision-making in complex environments.Source: §A.3 · p. 19
A.4 Plan Tips.
This section describes OAgents’ planning guidance and use of hierarchical memory and test-time search to support agent reasoning, including prompts for retaining execution history and assessing and reflecting on search steps.
A.4 Plan Tips.
- Experience-derived plan tips are distilled from past trajectories and injected as soft preferences, helping planners avoid known pitfalls and build more robust strategies.The tips are intended to guide action and subgoal selection in complex problem-solving and dynamic, uncertain environments.Source: §A.4 · p. 19
A.5.1 Current Memory.
- Current memory buffers recent observations and actions in temporal order, retaining fine-grained task information for on-the-fly decisions.It serves as a short-term buffer for dynamic environmental inputs.Source: §A.5.1 · p. 19
A.5.2 Memory Summarization.
- Memory summarization converts raw experience sequences into structured semantic units and extracts high-salience knowledge for downstream processing.The component uses topic modeling and sequence-to-sequence generation.Source: §A.5.2 · p. 19
A.5.3 Vectorized Memory Retrieval.
- Vectorized memory retrieval embeds execution logs in a shared latent space and retrieves historical memories by vector similarity.Retrieved memories are intended to be beneficial to the agent’s current task.Source: §A.5.3 · p. 20
A.5.4 Long-Term Memory.
- Long-term memory fuses current and existing long-term memory to carry insights from historical reasoning into task execution.It is designed to address lengthy reasoning chains and redundant context while generating optimization recommendations.Source: §A.5.4 · p. 20
A.6.1 Diversity Enhancement.
- Test-time scaling broadens reasoning paths by sampling across multiple agent policies, using differences in LLM capabilities to increase the range of candidate solutions.The module is organized around diversity, optimization, and reward modeling.Source: §A.6 · p. 20, §A.6.1 · p. 20
A.6.2 Optimization; A.6.3 Reward Modeling.
- Process-based rewards assess task progression, error handling, and efficiency at each step, while reflection feeds corrective insights into later reasoning for iterative refinement.These mechanisms are intended to improve convergence toward accurate responses and enable on-the-fly error correction.Source: §A.6.2 · p. 20, §A.6.3 · p. 20
B Prompts.
- The prompts direct planners to produce high-level plans, guide search agents to analyze and revise queries, and instruct memory operations to summarize execution and retrieve similar historical steps.Planning prompts also distinguish cohesive tasks from those that can be divided into subtasks.Source: §B.1 · pp. 21–22, §B.2 · p. 23, §B.3 · p. 25
B.3 Memory Prompts
- Memory prompts combine current and previous execution content with existing long-term memory, summarize the current step, retrieve relevant history, and update an ongoing long-term record.The summary is point-by-point and includes optimization suggestions.Source: §B.3 · p. 25
B.4 Test-Time Scaling Prompts
- The PRM-score prompt rates an action step on goal progress, errors and stability, efficiency, reflection use, loops, and consistency with prior planning and task steps.It requests valid JSON containing analysis and an integer score from 0 to 10.Source: §B.4 · p. 26
- The PRM-list prompt selects the trajectory that best advances the task, favoring efficient progress while penalizing real loops, errors, and incoherent planning.It considers value relative to trajectory depth, distinguishes valueless identical-node loops from repetitions that add progress, and assesses final-node proximity to the goal.Source: §B.4 · p. 27
- The single-node reflection prompt summarizes outcomes, checks success and score alignment, assesses confidence, and suggests focused next-attempt improvements grounded in provided fields.Suggestions should be conservative, limited to clear issues or opportunities, and minimal or absent when no significant issue is identified.Source: §B.4 · p. 28