PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenHands: An Open Platform for AI Software Developers as Generalist Agents

Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, Graham Neubig

arXiv:2407.16741v3cs.SEcs.AIcs.CL

TL;DR

Developing AI agents that can modify complex software, gather information, and operate safely remains challenging. OpenHands addresses this with a generalist agent platform, whose unchanged CodeAct agent performs competitively across software development, web interaction, and miscellaneous tasks.

  • Problem

    Building agents that modify complex software, gather information dynamically, and avoid harmful side effects remains challenging.

  • Method

    OpenHands provides a platform combining software interaction, sandboxed code execution, web browsing, multi-agent delegation, and broad evaluation capabilities.

  • Results

    The same unchanged CodeAct agent performs competitively across software development, web interaction, and miscellaneous tasks.

  • Takeaways & Limitations

    OpenHands supports research and applications involving agentic AI systems through reusable interaction, collaboration, safety, and evaluation infrastructure.

  • Takeaways & Limitations

    OpenHands has not yet reached 100% on HumanEvalFix, though the authors consider that benchmark performance feasible in future iterations.

Abstract

from arXiv · show

Software is one of the most powerful tools that we humans have at our disposal; it allows a skilled programmer to interact with the world in complex and profound ways. At the same time, thanks to improvements in large language models (LLMs), there has also been a rapid development in AI agents that interact with and affect change in their surrounding environments. In this paper, we introduce OpenHands (f.k.a. OpenDevin), a platform for the development of powerful and flexible AI agents that interact with the world in similar ways to those of a human developer: by writing code, interacting with a command line, and browsing the web. We describe how the platform allows for the implementation of new agents, safe interaction with sandboxed environments for code execution, coordination between multiple agents, and incorporation of evaluation benchmarks. Based on our currently incorporated benchmarks, we perform an evaluation of agents over 15 challenging tasks, including software engineering (e.g., SWE-BENCH) and web browsing (e.g., WEBARENA), among others. Released under the permissive MIT license, OpenHands is a community project spanning academia and industry with more than 2.1K contributions from over 188 contributors.

1 INTRODUCTION

OpenHands is introduced as a community-driven platform for developing generalist and specialist AI agents that interact with the world through software. It combines flexible agent–environment interaction, sandboxed execution, multi-agent delegation, and broad evaluation support in an MIT-licensed implementation.

  • Motivation: Open-source agent development is challenging because capable agents require interfaces, environments, and interaction mechanisms for development and evaluation.These frameworks commonly provide interfaces such as function calls or code execution, operating environments, and interaction mechanisms.
  • Motivation: Software provides a powerful interface for agents to interact with the world, but enabling agents to develop software introduces unique challenges.The paper motivates software as an interface because it supports complex real-world activities and has extensive development tooling.
  • OpenHands platform: OpenHands provides an event-stream architecture, a Docker-sandboxed runtime, software-engineer-like interaction, multi-agent delegation, and an evaluation framework.Agents can create and edit complex software, execute arbitrary sandboxed code, and browse websites to collect information; the framework evaluates agents across diverse tasks.
  • OpenHands platform: OpenHands includes an immediately usable implementation with over 10 agents, including a CodeAct-based generalist agent and specialist agents for web browsing and code editing.The platform is not only conceptual; it includes implemented agents, environments, and evaluations, with real-time user feedback.
  • Community: 32K GitHub stars and more than 2.1K contributions from over 188 contributors demonstrate OpenHands’s community traction under the permissive MIT license.The project spans academia and industry and permits commercial use.

2 OPENHANDS ARCHITECTURE

OpenHands structures agents around state, event streams, general actions, and a sandboxed runtime for code and browser interaction. Its architecture also supports extensible skills and delegation between specialized agents.

  • Agent definition: Agents perceive environmental state, including prior actions and observations, and produce actions while solving user-specified tasks.
  • State and Event Stream: The state records a chronological event stream of actions, observations, and user interactions, alongside auxiliary execution information such as accumulated LLM-call cost.
  • Actions: OpenHands provides general actions for arbitrary Python and bash execution in a sandbox and browser interaction through a domain-specific language.
  • Agent Runtime: Each task runs in a securely isolated Docker container connected through a REST API that executes event-stream actions and returns results as observations.
  • Skills and Multi-Agent Composition: AgentSkills packages reusable Python tools for automatic Jupyter import, while AgentDelegateAction lets agents delegate subtasks to specialized agents such as BrowsingAgent.

3 AGENTHUB: A HUB OF COMMUNITY-CONTRIBUTED AGENTS

OpenHands provides a community hub of agent implementations that users can select as task baselines. It includes generalist, web-browsing, graph-based multi-agent, and task-specialized micro-agent designs.

  • Community-contributed agents: OpenHands supports community-contributed agent implementations for end users and as baselines for different agent tasks.This support is based on the platform’s agent abstraction.
  • Generalist and browsing agents: CodeActAgent is the default generalist agent, combining natural-language communication with executable bash, Python, and browser-specific code.Its general action space supports tasks including software engineering and web interaction.
  • Generalist and browsing agents: Browsing Agent is a simple, effective zero-shot web-agent baseline with improved observations and actions relative to a WebArena-like design.The full prompts are provided in §K.
  • Multi-agent systems: GPTSwarm Agent uses optimizable graphs in which nodes perform operations and edges define collaboration and communication for multi-agent systems.The framework automatically optimizes nodes and edges.
  • Specialized agents: Micro agents specialize existing generalist-agent implementations for particular tasks while lowering development barriers through shareable specialized prompts.They reuse most implementations from agents such as CodeAct Agent.

4 EVALUATION

OpenHands integrates 15 established benchmarks spanning software engineering, web browsing, and miscellaneous assistance, comparing its agents with reproducible open-source baselines. The same CodeAct agent, without system-prompt modifications, performs competitively across all three categories, including 79.3% on HumanEvalFix and competitive WebArena performance.

  • Evaluation setup: 15 established benchmarks cover software engineering, web browsing, and miscellaneous assistance in OpenHands.The evaluation compares OpenHands with open-source reproducible baselines without benchmark-specific manual prompt engineering.
  • Overall results: The same CodeAct agent, without system-prompt modifications, demonstrates competitive performance across software development, web interaction, and miscellaneous tasks.OpenHands agents are designed for generality rather than top performance in every category.
  • Software engineering: 79.3% of HumanEvalFix bugs were successfully fixed by the OpenHands CodeActAgent.The evaluation uses the Python subset and allows self-debugging over multiple turns with feedback from test execution.
  • Software engineering: Achieving 100% on HumanEvalFix is considered feasible in future OpenHands iterations.The paper identifies this as a future objective after the reported result.
  • Web browsing: OpenHands’s BrowsingAgent achieves competitive performance on WebArena among agents using LLMs with domain-general prompting techniques.WebArena contains 812 human-curated tasks across shopping, forums, developer platforms, and content management systems.

5 CONCLUSION

OpenHands is a community-driven platform for developing agents that interact with the world through software interfaces. It combines interaction mechanisms, sandboxing, agent skills, multi-agent collaboration, and evaluation to support agentic AI research and applications.

  • 5 CONCLUSION: OpenHands enables development of agents that interact with the world through software interfaces.The platform is community-driven.
  • 5 CONCLUSION: The platform provides powerful interaction mechanisms, safe sandboxed environments, essential agent skills, multi-agent collaboration, and comprehensive evaluation.These capabilities are presented as core components of OpenHands.
  • 5 CONCLUSION: These capabilities accelerate research innovations and real-world applications of agentic AI systems.The conclusion frames OpenHands as supporting both research and practical applications.
  • 5 CONCLUSION: Developing safe and reliable agents remains challenging.The passage explicitly notes these challenges in §A.

A LIMITATIONS AND FUTURE WORK · B ETHICS STATEMENT · C RELATED WORK

OpenHands identifies future work spanning stronger agents, multimodal and file-editing capabilities, improved browsing, and less handcrafted workflow design. Its ethics approach emphasizes evaluation, human oversight, and access for safety research, while related work situates it among generalist, specialized, and software-development agent frameworks.

  • A LIMITATIONS AND FUTURE WORK: Future work aims to support multimodal interaction through standard IPython and browser integration, including images, videos, and XLSX processing.
  • A LIMITATIONS AND FUTURE WORK: OpenHands agents still struggle with complex tasks, motivating improvements through training and inference-time techniques.
  • A LIMITATIONS AND FUTURE WORK: Improving long-file editing is a priority because current agents suffer substantially when modifying long files.
  • A LIMITATIONS AND FUTURE WORK: OpenHands plans optional browsing components using Auto Eval & Refine, retry-on-error Reflexion prompts, and task-completion reward models.
  • A LIMITATIONS AND FUTURE WORK: Current workflows require substantial handcrafted work, motivating graph-based alternatives such as GPTSwarm and LangGraph for optimization methods including reinforcement learning and meta-prompting.
  • B ETHICS STATEMENT: OpenHands addresses risks from increasingly deployed agents by enabling systematic evaluation before widespread deployment.
  • B ETHICS STATEMENT: It also emphasizes human-agent interaction with oversight and worldwide researcher access to agent suites for frontier safety research.
  • C RELATED WORK: Related work includes generalist agent proposals, foundational and orchestration frameworks, browser-focused systems, prompt-optimization tools, and software-development agents.

D GRAPHICAL USER INTERFACE

OpenHands provides a graphical interface that visualizes the agent’s ongoing actions and supports real-time user feedback, including interruption and additional instructions.

  • D GRAPHICAL USER INTERFACE: The graphical interface visualizes actions such as web browsing and executing shell commands or Python code.It runs alongside the command-line interface and connects directly with the event streams.
  • D GRAPHICAL USER INTERFACE: Users can interrupt the agent at any moment to provide feedback, comments, or instructions while it works.

E QUALITY CONTROL: INTEGRATION TESTS FOR AGENTS · F HOW OPENHANDS RUNTIME WORK

OpenHands uses end-to-end integration tests to detect agent regressions without repeatedly running expensive full evaluations, while its runtime workflow is presented as a central part of the platform. The framework combines task-result verification, deterministic LLM mocking, response regeneration for breaking changes, and automated multi-platform testing.

  • E QUALITY CONTROL: INTEGRATION TESTS FOR AGENTS: Full-suite evaluations measure performance degradation, but running them after every code change can be prohibitively slow and expensive.The paper motivates integration tests as a faster quality-control mechanism for complex agents, where minor development errors can hurt final task performance.
  • E QUALITY CONTROL: INTEGRATION TESTS FOR AGENTS: Integration tests validate OpenHands end-to-end by automating task execution and comparing outputs with predefined expected results or a gold file.Developers specify tasks such as correcting typos in “bad.txt,” then verify the resulting output against the corresponding gold file.
  • E QUALITY CONTROL: INTEGRATION TESTS FOR AGENTS: Predefined responses matched to exact prompts make integration tests deterministic while reducing reliance on costly real LLM calls.The framework intercepts all LLM calls and supplies stored responses for exact prompt matches.
  • E QUALITY CONTROL: INTEGRATION TESTS FOR AGENTS: Breaking prompt or task-handling changes trigger regeneration of stored prompt-response pairs with real LLMs, while routine tests reuse existing responses after slight prompt adjustments.A script creates and stores pairs for new tests or modified prompts, preserving reusable responses when changes are minor.
  • E QUALITY CONTROL: INTEGRATION TESTS FOR AGENTS: Automated tests run for every pull request and main-branch commit across multiple platforms, environments, sandboxes, and agents.Supported configurations include Linux and Mac, plus local, SSH, and exec sandboxes.
  • E QUALITY CONTROL: INTEGRATION TESTS FOR AGENTS: Stored prompt-response pairs support prompt regression testing, change tracking, and onboarding by showing new team members how LLM interactions work.The stored pairs provide a reference for understanding interactions and detecting regressions.
  • E QUALITY CONTROL: INTEGRATION TESTS FOR AGENTS: 600 USD is the approximate cost of running a SWE-Bench Lite evaluation with gpt-4o.This cost illustrates why repeatedly running full evaluations can be impractical during development.
  • F HOW OPENHANDS RUNTIME WORK: Figure 4 presents the OpenHands runtime workflow.The figure is identified as the platform’s runtime workflow.

F.1 WORKFLOW · F.2 HOW OPENHANDS BUILDS AND MAINTAINS RUNTIME IMAGES

OpenHands uses Docker-based client-server workflows to build customized runtime images, launch sandboxed containers, and exchange actions and observations through a RESTful runtime client. Its image-management approach targets efficiency, consistency, and flexibility across production and development environments.

  • F.1 WORKFLOW: The workflow uses a Docker-based client-server architecture for runtime interaction.The runtime system is implemented with Docker containers.
  • F.1 WORKFLOW: OpenHands accepts a user-provided base Docker image and builds an OH runtime image from it.The OH runtime image includes OpenHands-specific code, primarily the runtime client.
  • F.1 WORKFLOW: At startup, OpenHands launches a Docker container using the OH runtime image.
  • F.1 WORKFLOW: The backend communicates with the runtime client over a RESTful API, sending actions and receiving observations.
  • F.1 WORKFLOW: The runtime client safely executes shell commands, file operations, Python code, and other actions inside the sandboxed container.
  • F.1 WORKFLOW: The client intermediates between the backend and sandbox, manages environment state, and formats execution results into consistent observations.Managed state includes the current working directory and loaded plugins.
  • F.2 HOW OPENHANDS BUILDS AND MAINTAINS RUNTIME IMAGES: OpenHands’ runtime-image approach is designed to provide efficiency, consistency, and flexibility.These goals apply to creating and maintaining Docker images for both production and development environments.

F.2.1 IMAGE TAGGING SYSTEM

OpenHands uses dual tags for runtime images, combining hash-based reproducibility with generic tags that reference the latest build for a configuration. This supports efficient management of development and production environments.

  • Dual-tagging approach: The dual-tagging approach balances reproducibility with flexibility and efficiently manages development and production environments.Hash-based tags ensure reproducibility, while generic tags provide stable references to the latest configured version.
  • Hash-based tag: Hash-based tags use the MD5 hash of the Docker build folder, including runtime source code, dependencies, and Dockerfile.Identical hash tags guarantee identical source code and Dockerfile, ensuring reproducibility and consistent image contents.
  • Generic tag: Generic tags follow the runtime:oh_v{VERSION}_{BASE_IMAGE}_tag_{IMAGE_TAG} format and identify the latest build for a base-image and OpenHands-version combination.The generic tag is updated whenever a new image is built from the same base image.

F.2.2 BUILD PROCESS

OpenHands builds runtime images by hashing a generated build context, reusing compatible images when available, and falling back to a fresh build when necessary. New images receive both hash-based and generic tags to support later reuse.

  • Image Naming and Build Preparation: The system converts a user-provided base image, such as ubuntu:22.04, into an OpenHands runtime image name, such as runtime:oh_v0.9.3_ubuntu_tag_22.04.It generates a build context containing the Dockerfile and OpenHands source code, then calculates its hash.
  • Image Selection: The build process checks for an image matching the calculated hash before looking for a recent compatible image to use as a base.If no compatible image exists, it builds from scratch using the original base image.
  • Image Reuse and Rebuilding Logic: An existing hash-tagged image is reused as is; otherwise, the system rebuilds using the latest generic image as a base to leverage existing dependencies.This reuse strategy saves build time by avoiding unnecessary dependency installation.
  • Fallback and Tagging: If neither hash-tagged nor generic-tagged images are found, the system builds completely from scratch and tags the new image with both tag types.The hash-based tag has the form target_image_repo:target_image_hash_tag, while the generic tag has the form target_image_repo:target_image_tag.

G ADDITIONAL RESULTS FOR GPQA BENCHMARK … K BROWSING AGENT DETAILS

The appendix expands GPQA evaluation results and documents OpenHands’ supported agent skills, BrowserGym action primitives, and browsing-agent prompting procedures. These details cover available operations, sequential action prediction, and concise answer formatting for exact-match evaluation.

  • G ADDITIONAL RESULTS FOR GPQA BENCHMARK: Additional GPQA results report performance on other benchmark subsets in Table 7.The table is titled “Full Evaluation Results on the GPQA Benchmark.”
  • I SUPPORTED AGENTSKILLS: As of OpenHands v0.6, the platform supports a documented list of agent skills maintained in the source code.The listed skills include file editing, search, document parsing, and multimedia parsing operations.
  • J BROWSERGYM ACTIONS: BrowserGym actions are grouped into agent-control, navigation, page-element, coordinate-based, and tab-related categories.OpenHands uses BrowserGym v0.3.4 actions as its main browsing action primitives, with configurable subsets of functionality.
  • K BROWSING AGENT DETAILS: The browsing prompt provides the current page state and task information so the agent can predict the best next action.The prompt includes the goal, accessibility tree, and previous actions, and requires responses in a program-interpretable format.
  • K BROWSING AGENT DETAILS: Multiple actions may be predicted in one turn when they are intended to execute sequentially without page feedback.This can reduce turns for workflows such as filling a username and password, but actions requiring feedback should not be combined.
  • K BROWSING AGENT DETAILS: 16 different types of actions are available to the browsing agent, including messaging, scrolling, form filling, selection, clicking, dragging, and file uploading.The examples enumerate actions such as fill, select_option, click, dblclick, press, drag_and_drop, and upload_file.
  • K BROWSING AGENT DETAILS: For WebArena exact-match tasks, the agent is instructed to reply with only a concise answer string when messaging the user.The instruction is intended to prevent extra text from causing evaluation failure.
Loading 2407.16741v3…