PaperlayerView exact revision on arXiv

Source-linked AI summary

Modeling Distinct Human Interaction in Web Agents

Faria Huq, Zora Zhiruo Wang, Zhanqiu Guo, Venu Arvind Arangarajan, Tianyue Ou, Frank Xu, Shuyan Zhou, Graham Neubig, Jeffrey P. Bigham

arXiv:2602.17588v4cs.CLcs.HC

TL;DR

Web agents need to anticipate human intervention because users correct errors and align behavior with preferences, while current systems lack a principled model of intervention timing. The paper collects COWCORPUS, characterizes collaboration styles, and trains intervention-aware models for stepwise prediction. These models improve intervention prediction and increase user-perceived usefulness when deployed in live agents.

  • Problem

    Current web agents lack a principled understanding of when and why humans intervene during task execution.

  • Method

    The paper collects 400 real-user collaborative web trajectories, identifies four interaction styles, and fine-tunes multimodal models to predict intervention from task context.

  • Results

    61.4–63.4% intervention prediction accuracy improvement over base LMs and a 36.8% increase in user-perceived usefulness are reported for intervention-aware agents.

  • Takeaways & Limitations

    Structured modeling of intervention supports agents that adapt to human preferences and collaboration styles over time.

  • Takeaways & Limitations

    PTS uses α=0.2 by default, with sensitivity tested over α values from 0.1 to 0.5.

Abstract

from arXiv · show

Despite rapid progress in autonomous web agents, human involvement remains essential for shaping preferences and correcting agent behavior as tasks unfold. However, current agentic systems lack a principled understanding of when and why humans intervene, often proceeding autonomously past critical decision points or requesting unnecessary confirmation. In this work, we introduce the task of modeling human intervention to support collaborative web task execution. We collect CowCorpus, a dataset of 400 real-user web navigation trajectories containing over 4,200 interleaved human and agent actions. We identify four distinct patterns of user interaction with agents -- hands-off supervision, hands-on oversight, collaborative task-solving, and full user takeover. Leveraging these insights, we train language models (LMs) to anticipate when users are likely to intervene based on their interaction styles, yielding a 61.4-63.4% improvement in intervention prediction accuracy over base LMs. Finally, we deploy these intervention-aware models in live web navigation agents and evaluate them in a user study, finding a 36.8% increase in user-rated agent usefulness. Together, our results show structured modeling of human intervention leads to more adaptive, collaborative agents.

1. Introduction

Web agents still depend on human intervention, but existing systems do not model when or why users intervene. This work introduces COWCORPUS and intervention-aware models to support collaboration adapted to user interaction styles.

  • Motivation: Existing agents may continue under incorrect assumptions about user intent or request confirmation at inappropriate moments.Human involvement remains necessary to correct misinterpretations and realign agent behavior with preferences.
  • Motivation: Prior collaborative-agent approaches focus on specific mechanisms rather than the broader spectrum of mid-task intervention, alternative action-taking, and control transfer.
  • Dataset and interaction styles: COWCORPUS contains 400 real-user collaborative web trajectories with 2,748 agent action steps and 1,476 human action steps.Annotations mark interventions involving pausing, resuming, or overriding agent execution.
  • Dataset and interaction styles: Analysis identifies four collaboration styles—Takeover, Hands-on, Hands-off, and Collaborative—based on recurring intervention and control-sharing patterns.These styles reflect needs including error correction, preference refinement, and assistive takeover.
  • Intervention-aware modeling: 61.4–63.4% intervention prediction accuracy improvement over baselines is achieved by intervention-aware models across multiple model backbones.
  • Evaluation: 36.8% increase in user-perceived usefulness over baseline systems is achieved when intervention-aware models are deployed in live web agents.The authors frame this result as evidence of more adaptive human–agent collaboration in practice.

2. Problem Formulation: Human Intervention Modeling

The paper formulates human intervention prediction within a partially observable web-task process. Its model predicts step-level intervention from trajectory context, current observations, and the proposed agent action, while PTS evaluates timing as well as correctness.

  • Problem formulation: Human–agent collaboration is formulated as a POMDP with agent and human policies acting over multimodal web observations.Each observation combines the current screenshot and webpage accessibility tree with the past trajectory.
  • Problem formulation: Human intervention is represented as a binary variable and modeled through step-wise binary classification.The predictor estimates whether the human will intervene at each time step.
  • Modeling approach: The supervised-fine-tuned multimodal model receives trajectory history, the current observation, and the agent-proposed action, then outputs intervention or continuation tokens.
  • Evaluation metrics: Step Accuracy, F1 Score, and Perfect Timing Score evaluate intervention prediction across trajectory steps.Step Accuracy measures intervention correctness, while F1 captures the harmonic mean of precision and recall.
  • Evaluation metrics: PTS penalizes false-positive intervention predictions according to their temporal distance before the ground-truth intervention.The score is normalized to [0, 1], with higher values indicating more accurate and well-timed predictions.
  • Evaluation metrics: PTS uses α=0.2 by default and remains consistent across tested α values from 0.1 to 0.5.

3. COWCORPUS: Collecting Human-Agent Collaborative Web Activities

COWCORPUS captures collaborative web activity through standardized and user-selected tasks, recording interleaved human-agent actions and intervention behavior. The analysis identifies recurring intervention motives and four stable interaction patterns that differ in timing, intensity, and control sharing.

  • 3.1. Data Collection: COWCORPUS combines 10 Mind2Web standard tasks with 10 participant-chosen free-form tasks, recording actions, actors, timestamps, and web snapshots from 20 users.The dataset design balances controlled comparison with unconstrained, user-driven objectives.
  • 3.1. Data Collection: The corpus evaluates human and agent step counts and execution time across both standardized and free-form collaborative sessions.Table 3 summarizes intervention intensity, actor step counts, and time.
  • 3.2. Why Do Users Intervene?: Users intervene to correct premature or looping agent behavior, address preference misalignment, and compensate for complex web environments.Preference issues can emerge mid-task when initial instructions are incomplete or underspecified.
  • 3.2.2. Interaction Patterns: Four user groups emerge: Takeover users intervene late and retain control; Hands-on users intervene frequently while alternating control; Hands-off users rarely intervene; Collaborative users intervene selectively and return control.These groups differ systematically in intervention frequency, intensity, timing, and handback behavior.
  • 3.2.2. Interaction Patterns: The interaction patterns are consistent across tasks and motivate models that represent distinct human-agent collaboration styles.The clustering uses four participant-level intervention features and k-means with k=4.

4. Experiments: Modeling Human Intervention

The experiments model intervention as a stepwise prediction problem and compare general and style-conditioned models with prompting, fine-tuned, and extreme-policy baselines. Fine-tuning on COWCORPUS improves timing-aware prediction, while cluster-specific models generally perform best for their corresponding interaction styles.

  • 4. Experiments: The study trains a general intervention-aware model and style-conditioned models tailored to distinct collaboration preferences.Models are evaluated against prompting-based proprietary LMs, fine-tuned open-weight models, and non-learning baselines.
  • 4.2. Benchmarking Intervention Awareness: The evaluation uses trajectory-level splits, intervention-ratio consistency, and step accuracy, F1, and Perfect Timing Score metrics.Hands-off trajectories are excluded because they contain no intervention events.
  • 4.2. Benchmarking Intervention Awareness: GPT-4o achieves Non-interv F1 0.846 but Interv F1 0.198, producing a PTS of 0.147 and indicating difficulty predicting active interventions.The reported disparity reflects strong performance on non-intervention steps but weak intervention detection.
  • 4.2. Benchmarking Intervention Awareness: Gemma-27B (SFT) achieves PTS 0.303, exceeding Claude 4 Sonnet at 0.293, while LLaVA-8B (SFT) reaches PTS 0.201 and exceeds GPT-4o at 0.147.The paper reports the strongest gains for open-weight models fine-tuned on COWCORPUS.
  • 4.2. Benchmarking Intervention Awareness: The Always No Interv baseline has 85.3% step accuracy but PTS 0, whereas Always Interv has PTS 0.151, showing that timing matters beyond binary classification.The two policies represent fully autonomous and fully confirmation-dependent extremes.
  • 4.3. Interaction Pattern Customization: Cluster-specific models generally outperform other models on their corresponding interaction clusters, except that the Hands-on model performs best for Takeover users.The exception is attributed to data sparsity in the Takeover cluster, which contains only 11 intervention steps out of 131.

5. Deploying Collaborative Web Agents

The intervention-aware model is deployed in PLOWPILOT, which prompts users only when intervention is predicted to be likely. In a follow-up study, the intervention-aware agent receives higher user ratings than the baseline system.

  • 5. Deploying Collaborative Web Agents: PLOWPILOT integrates intervention modeling into a Chrome-based web navigation agent and requests user input only at predicted high-likelihood intervention moments.This replaces confirmation or intervention opportunities at every step with selective prompting.
  • 5. User Study: 36.8% higher user ratings are reported for the intervention-aware agent compared with existing collaborative agents.The comparison is summarized in the post-study Likert questionnaire results.
  • 5. User Study: The follow-up evaluation combines four returning annotators with 12 new participants, yielding a reported sample of n=16.Participants completed 10 standard and 10 free-form tasks and used cluster-customized models.
  • 5. User Study: The questionnaire measures task effort, intervention effort, perceived automation benefits, control, preference alignment, and task speed.Ratings use a 7-point Likert scale across six dimensions.

6. Related Work

Related work spans autonomous web agents and human-AI collaboration frameworks. The paper positions its contribution as extending web-agent systems toward interactive control and broader modeling of human intervention patterns.

  • Autonomous Web Agents: Web-agent benchmarks and computer-use systems have advanced navigation across complex, real-world, multi-domain tasks.The cited systems use representations such as HTML and accessibility trees or support computer-use interaction.
  • Autonomous Web Agents: Existing autonomous extensions often prioritize autonomy over collaboration and lack mechanisms for interactive user control.This paper emphasizes human-agent collaboration beyond solo agent autonomy.
  • Modeling Human-Agent Collaboration: Human-AI collaboration research includes robotics, productivity tools, and LLM-based frameworks for co-planning and co-execution.The related frameworks cover real-time web and persistent collaborative settings.

7. Conclusion

The paper models human intervention as a structured behavioral signal reflecting distinct collaboration styles, then deploys intervention-aware agents that improve prediction accuracy and user satisfaction.

  • 61.4–63.4% improvement in intervention prediction accuracy over base LMs demonstrates the value of fine-tuning on collaborative interaction traces.The models capture when users choose to intervene during web navigation.
  • 36.8% increase in user satisfaction shows tangible benefits when live web agents anticipate human intervention.The evaluation used a live web agent.
  • Human intervention reflects distinct collaboration styles ranging from passive supervision to active co-piloting.COWCORPUS contains 400 real-user web navigation trajectories for studying these patterns.

A.1. Task Annotation Setup and Participant Information

COWCORPUS was collected through collaborative web tasks in CowPilot, combining benchmark and participant-chosen tasks while recording human-agent actions and web states.

  • Task Annotation Setup and Participant Information: Participants executed 20 web tasks collaboratively with an LM-based agent and received $0.50 per task.Participants were aged 20–30 and had varied AI-agent familiarity and daily web-task experience.
  • Task Annotation Setup and Participant Information: Free-form tasks were constrained to multiple steps and varied in length and complexity.Annotators were familiar with agentic frameworks.
  • Task Annotation Setup and Participant Information: Participants could choose whether to share task logs after reviewing annotation summaries, preserving control over data disclosure.The data collection paradigm was intentionally self-initiated.
  • Task Annotation Setup and Participant Information: CowPilot proposes visually highlighted UI actions that users may approve, pause, or override at arbitrary times.The system logs the interaction between users, agents, and the web environment.

A.3.2. Current Agent Bottleneck: Time Demand

Collaborative execution adds time because agent actions are slower and users must monitor progress, while intervention-aware interaction can reduce unnecessary monitoring but does not clearly save overall time.

  • Current Agent Bottleneck: Time Demand: 93.1 seconds for standard tasks and 71.7 seconds for free-form tasks were average agent execution times in CowPilot.Human intervention averaged 23.9 seconds and 13.8 seconds for the same task types, respectively.
  • Current Agent Bottleneck: Time Demand: Users rated task-speed improvement neutrally at 4.05, indicating uncertainty about whether CowPilot saved time versus working unaided.The rating came from post-annotation feedback.
  • Current Agent Bottleneck: Time Demand: Agent latency accumulates in longer-horizon tasks because agents proceed at a constant pace unless interrupted.Continuous monitoring also requires users to maintain observation windows for inspecting each step.
  • Current Agent Bottleneck: Time Demand: 5.25 average time requirement and 5.75 avoidance-of-unnecessary-interruption rating favored PLOWPILOT’s need-based interaction module.Users were more satisfied when the agent intervened only as needed rather than requiring continuous monitoring.
  • Current Agent Bottleneck: Time Demand: Current agents cannot proactively request help, motivating uncertainty estimation for identifying decision points where user input is most valuable.This would replace a fixed execution pace with more targeted help-seeking.

B.1. Ablation: Few Shot Example Count

Few-shot prompting affects intervention timing inconsistently across models, while explicit reasoning tends to reduce perfect timing despite slightly improving step accuracy.

  • Few Shot Example Count: 2-shot prompts slightly improve PTS for GPT-4o and Gemini but significantly degrade Claude’s intervention timing.The evidence suggests zero-shot prompting can be more robust for this task in some models.
  • Models with Reasoning vs No Reasoning: Explicit reasoning tends to lower Perfect Timing Score across models.Reasoning slightly improves Step Accuracy by reducing false positives but strengthens models’ tendency to remain silent.

B.3. Ablation: Impact of Human Action History

Ablations show that intervention prediction benefits from human action history and multimodal observations, while PTS preserves model rankings across temporal-penalty settings. These results support using interaction trajectories and stable evaluation metrics when assessing collaborative agents.

  • Human Action History: 81.36% Step Accuracy with human action history exceeds 76.27% without it, showing that past interactions improve intervention prediction.The ablation removes human action history from the agent’s input to isolate temporal context.
  • Input Format: 74.9% multimodal Step Accuracy exceeds Screenshot-only 72.9% and AXTree-only 69.7%, demonstrating the benefit of combining visual and structural inputs.Both single-modality approaches perform below the multimodal baseline.
  • PTS Stability: Across α values, PTS decreases monotonically for zero-shot baseline models while preserving their relative ranking, as confirmed by high Kendall’s W.The metric remains stable for comparing models despite changes in the temporal penalty.
  • PTS Stability: Fine-tuned models show flatter PTS curves across α, indicating temporally accurate predictions with fewer premature or missed intervention calls.Because α controls penalties for early or mistimed predictions, small changes in fine-tuned scores imply limited timing penalties.
Loading 2602.17588v4…