Source-linked AI summary
CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation
Faria Huq, Zora Zhiruo Wang, Frank F. Xu, Tianyue Ou, Shuyan Zhou, Jeffrey P. Bigham, Graham Neubig
TL;DR
Web agents can struggle with complex tasks, while evaluating how humans and agents collaborate remains challenging. COWPILOT lets them alternate actions and evaluates task success and collaboration; its collaborative mode reports higher success than autonomous and human-only settings across five websites. The framework is also presented as a tool for collecting trajectories and evaluating agents.
Problem
Although LLMs can automate digital tasks, measuring human-agent collaboration and the influence of user interventions remains challenging.
Source: §1 · p. 1Method
COWPILOT lets an LLM propose actions that users may pause or reject, take over, and later resume, and provides metrics for evaluating collaboration.
Source: §1 · p. 1Results
47% higher success rates over autonomous agents and 6% over human-only settings were reported for collaborative mode across five common websites.
Source: §1 · pp. 1–2Takeaways & Limitations
COWPILOT is presented as a tool for web automation, trajectory and feedback collection, and agent evaluation research.
Source: §1 · pp. 1–2COWPILOT currently requires a human observer, and results and intervention patterns may vary by user.
Source: §6 · p. 7
Abstract
from arXiv · showhide
While much work on web agents emphasizes the promise of autonomously performing tasks on behalf of users, in reality, agents often fall short on complex tasks in real-world contexts and modeling user preference. This presents an opportunity for humans to collaborate with the agent and leverage the agent's capabilities effectively. We propose CowPilot, a framework supporting autonomous as well as human-agent collaborative web navigation, and evaluation across task success and task efficiency. CowPilot reduces the number of steps humans need to perform by allowing agents to propose next steps, while users are able to pause, reject, or take alternative actions. During execution, users can interleave their actions with the agent by overriding suggestions or resuming agent control when needed. We conducted case studies on five common websites and found that the human-agent collaborative mode achieves the highest success rate of 95% while requiring humans to perform only 15.2% of the total steps. Even with human interventions during task execution, the agent successfully drives up to half of task success on its own. CowPilot can serve as a useful tool for data collection and agent evaluation across websites, which we believe will enable research in how users and agents can work together. Video demonstrations are available at https://oaishi.github.io/cowpilot.html
1 Introduction
Web agents can automate complex tasks, but evaluating collaboration and the effects of user intervention remains challenging. COWPILOT supports alternating human-agent actions and reports higher task success in collaborative mode across five websites.
1 Introduction
- Prior frameworks mainly support natural-language feedback or record human-only actions, rather than dynamic alternation between humans and agents within a task.The paper asks how to enable collaborative task-solving and how agents perform in these settings.Source: §1 · p. 1
- COWPILOT lets users pause or reject proposed actions, take alternatives, and resume agent control during task execution.This alternating control is intended to ease human effort while allowing intervention.Source: §1 · p. 1
- COWPILOT also supports web automation, trajectory and feedback collection, and evaluation research across websites.The authors present it as a useful tool for future web automation, data collection, and agent evaluation.Source: §1 · pp. 1–2
- Across five common websites, the studies examined shopping, social, and technical domains.The supplied passage establishes the number of websites and the domains, but does not name individual sites.Source: §1 · p. 1
- 47% higher success rates over autonomous agents and 6% over human-only settings were achieved in COWPILOT’s collaborative mode.The LLM agent performed 84.8% of task steps and could drive up to half of task success.Source: §1 · p. 2
2 COWPILOT
COWPILOT alternates LLM and human actions in a web task, allowing the user to intervene and later return control to the agent. It pairs this interaction with metrics for task success and collaboration.
2.1 The COWPILOT System
- COWPILOT models a task as a sequence of actions by either an LLM policy or a human agent, based on the current observation and prior actions.The sequence continues until a task termination condition is reached.Source: §2.1 · p. 2
- Human actions provide corrections that enter the agent’s action history, helping it account for user input when generating subsequent actions.COWPILOT captures webpage and UI-element interactions and transforms raw human actions into the agent’s action space.Source: §2.1 · pp. 2–3
- The LLM proposes a next action for user review, and the user can reject or pause it, take over, and resume agent control later.A proposal is automatically executed if the user does not oppose it within five seconds; the take-over-and-return process can repeat without limit.Source: §2.1 · pp. 2–3
2.2 Evaluation Metrics
- The system measures end-to-end task accuracy by whether the objective is achieved, using the agent’s finish or failure action, which the user may override.The evaluation introduces metrics for task accuracy, user experience, and efficiency.Source: §2.2 · p. 3
- Collaboration metrics count agent, human, and total steps, interventions, and tasks completed with the agent taking the terminating step.Intervention count tracks user pauses; agent-driven completion accuracy indicates whether the agent can complete a task after human intervention.Source: §2.2 · pp. 3–4
3 Use Cases of COWPILOT
COWPILOT is presented as a tool for web automation and agent research, including trajectory collection and model evaluation. Its use cases span task execution, data annotation, and comparisons across models.
3 Use Cases of COWPILOT
- COWPILOT is proposed as a tool for agentic research on collaborative behavior, with three use cases in the paper’s scope.The listed applications include web automation, data collection, and evaluation.Source: §3 · p. 4
3.1 Web Automation
- As a web automation framework, COWPILOT supports fully autonomous task execution and a CoPilot mode in which people and agents collaborate.The Chrome extension handles computations locally apart from LLM calls and can use models available through LiteLLM.Source: §3.1 · p. 4
3.2 Data Collection from Websites
- COWPILOT collects task trajectories from Chrome-accessible websites, including simulated, sandboxed, and self-hosted sites.Collected human feedback can indicate whether a step is judged to lead to success and whether the full trajectory solves the task.Source: §3.2 · pp. 4–5
3.3 Evaluation and Comparative Analysis of Agent Performance
- The framework supports evaluation and comparison of open-weight and closed-source models, while this paper compares GPT and LLAMA.Users can redo a task with different models under identical conditions; the model backbone cannot be changed during a task.Source: §3.3 · p. 5
4 Exemplar Findings via COWPILOT
On a 27-task WebArena subset, CoPilot with GPT-4O achieved higher task accuracy than autonomous and human-only modes while requiring relatively few human steps. The study also reports that GPT-4O agents completed up to half of tasks in CoPilot mode, and describes when people intervened.
4 Exemplar Findings via COWPILOT
- The evaluation used 27 WebArena tasks across easy, medium, and hard levels, with three authors serving as human agents.A human-only baseline was also included.Source: §4 · p. 5
4.1 Copilot Mode Achieves the Best Accuracy
- 95% task accuracy with GPT-4O in CoPilot mode exceeded 48% in autonomous mode by 97.9% relatively and surpassed human task-solving accuracy by 6.7%.The evaluation averaged three authors’ task performance.Source: §4.1 · p. 5, §4 · p. 5
- In a WebArena subset evaluation of 27 tasks, copilot mode with the smaller LLAMA 8B model does not bring similar accuracy increases, but slightly degrades task accuracy by 8%.The evaluation considered fully autonomous and copilot settings, using gpt-4o-2024-08-06 and Llama-3.1-8B-Instruct as backbones.Source: §4.1 · p. 5, §4 · p. 5
4.2 Copilot Mode Requires Minimal Human Intervention
- GPT-4O required an average of 1.1 human steps, 15.2% of the trajectory, while the LLM agent performed 84.8% of task steps.With LLAMA, humans and agents spent roughly similar effort: 4.77 and 4.15 steps, respectively.Source: §4.2 · p. 5
- People often intervened when the LLM repeated invalid actions or made an obviously wrong choice, especially on unfamiliar or crowded layouts.One example was clicking “Customers” instead of “Orders” when searching for an order.Source: §4.2 · p. 5
4.3 Agents Drive Up to Half of the Success
- GPT-4O agent-driven completion accuracy reached up to 52% in CoPilot mode, with the agent initiating half of the successful tasks.The reported CoPilot task accuracy was 0.95.Source: §4.3 · p. 6
5 Related Works
Prior web-agent tools include APIs, isolated browser instances, and browser extensions, but existing extensions lack richer collaboration during tasks. COWPILOT supports dynamic, real-time human-agent interaction for practical web navigation.
5.1 Web Agent Plugin
- Dedicated Chromium instances can isolate browsing sessions, restrict multi-tab navigation, and diverge from standard workflows, limiting practical usability.API-based tools may require Docker setup, which poses barriers for non-technical users.Source: §5.1 · p. 6
- Browser extensions integrate into standard browsing environments, but the compared extensions lack features for richer human-agent collaboration.The comparison considers live and dynamic websites, human-only annotation, human-agent interaction and co-execution, and interactive agent evaluation.Source: §5.1 · p. 6
5.2 LLM Agents for Web automation
- Web automation has progressed from HTML and accessibility-tree agents to visual systems and benchmarks for complex multi-step tasks on e-commerce and real-world websites.The passage names prior systems and benchmarks including MiniWoB, WebShop [Yao et al., 2022], and WebArena [Zhou et al., 2023].Source: §5.2 · pp. 6–7
- COWPILOT addresses limited human-in-the-loop support with dynamic interaction, including suggest-then-execute, pause, and resume.The authors describe it as a platform for developing and evaluating agents in practical settings.Source: §5.2 · p. 7
6 Limitation and Future Work
COWPILOT currently depends on a human observer, and its comparative results and intervention patterns may vary with evaluation order and user behavior. The authors describe reducing observation demands and broadening evaluation as future work.
6 Limitation and Future Work
- COWPILOT currently requires a human observer, and future work would limit observation to critical steps rather than require constant oversight.The current setup is intentional for simulating task execution in a live setting.Source: §6 · p. 7
- The autonomous-versus-CoPilot comparison may have ordering bias, while results and intervention patterns may vary by user.A larger study across diverse demographics and websites is underway to assess and mitigate these behavioral patterns.Source: §6 · p. 7
- Future work would test a multi-LLM setup that uses a second LLM to simulate a user and approximate human decisions automatically.The authors also propose incorporating an active-learning framework.Source: §6 · p. 7
Societal Impact
The authors identify privacy, security, and unintended-action risks in web-agent deployment. They withhold the COWPILOT codebase pending an extensive user study, while releasing the Chrome extension package.
Societal Impact
- Tracking user actions may expose sensitive information, and agents may perform harmful or irreversible actions, such as confirming transactions without consent.The authors also warn that malicious actors could misuse COWPILOT.Source: Societal Impact · p. 7
- The authors have not open-sourced the codebase pending an extensive user study, and currently release only the final Chrome extension package.They identify safety risks and transparency as areas for future work.Source: Societal Impact · p. 7
- Users provide their own API key, and the authors state that users can modify the extension to use another LLM so their information is not shared with them.Future work could develop safeguards against unintended actions.Source: Societal Impact · p. 7
A Appendix
COWPILOT displays evaluation metrics after a task and lets users review or save the resulting trajectory data.
A Appendix
- After task completion, COWPILOT displays a summary of evaluation metric values.Source: A Appendix · p. 9
- Scores are automatically calculated from user-intervention counts and pause/resume statistics.Source: A Appendix · p. 9
- Users can modify summary entries and download a local copy of the trajectory data.Source: A Appendix · p. 9
A.1 Prompt for Action Transformation
The action-transformation prompt maps raw user events into the agent’s action space. Its output is used for action prediction when the agent resumes.
A.1 Prompt for Action Transformation
- It supplies the LLM with event-log and agent-action-space structures to represent the input action.Source: §A.1 · p. 9
- The LLM returns a list of actions with natural-language descriptions for agent action prediction after resumption.Source: §A.1 · p. 9
- The prompt transforms raw user actions into actions in the agent’s action space.Source: §A.1 · p. 9, Figure 5 · p. 10