Source-linked AI summary
When Users Change Their Mind: Evaluating Interruptible Agents in Long-Horizon Web Navigation
Henry Peng Zou, Chunyu Miao, Wei-Chieh Huang, Yankai Chen, Yue Zhou, Hanrong Zhang, Yaozu Wu, Liancheng Fang, Zhengyao Gu, Zhen Zhang, Kening Zheng, Fangxin Wang, Yi Nian, Shanghao Li, Wenzhe Fan, Langzhou He, Weizhi Zhang, Xue Liu, Philip S. Yu
TL;DR
Long-horizon web agents need to adapt when users change their intent, but existing evaluations provide limited evidence in environmentally constrained settings. The paper introduces InterruptBench and a trajectory-grounded framework for controlled interruption scenarios, evaluating six LLM backbones across single- and multi-turn interactions. Interruption handling remains a persistent bottleneck, with uneven and sometimes unstable success gains from additional updates.
Problem
Existing benchmarks assume uninterrupted execution or study interruptions in short, unconstrained language tasks, leaving long-horizon environmentally grounded interruption handling insufficiently studied.
Method
The paper formalizes addition, revision, and retraction scenarios, synthesizes semantically constrained WebArena-Lite tasks, and evaluates six LLM backbones with trajectory-grounded interruptions.
Results
Interruption handling remains a persistent bottleneck across six strong backbones, with additional updates generally improving success but producing uneven and sometimes unstable gains.
Takeaways & Limitations
Reliable interruptible agents require stronger mechanisms for state tracking, intent reconciliation, and error recovery during execution.
Takeaways & Limitations
The benchmark assumes interruption messages preserve the original task meaning and ground-truth answer, and that each interruption is necessary for solving the task.
Abstract
from arXiv · showhide
As LLM agents transition from short, static problem solving to executing complex, long-horizon tasks in dynamic environments, the ability to handle user interruptions, such as adding requirement or revising goals, during mid-task execution is becoming a core requirement for realistic deployment. However, existing benchmarks largely assume uninterrupted agent behavior or study interruptions only in short, unconstrained language tasks. In this paper, we present the first systematic study of interruptible agents in long-horizon, environmentally grounded web navigation tasks, where actions induce persistent state changes. We formalize three realistic interruption types, including addition, revision, and retraction, and introduce InterruptBench, a benchmark derived from WebArena-Lite that synthesizes high-quality interruption scenarios under strict semantic constraints. Using a unified interruption simulation framework, we evaluate six strong LLM backbones across single- and multi-turn interruption settings, analyzing both their effectiveness in adapting to updated intents and their efficiency in recovering from mid-task changes. Our results show that handling user interruptions effectively and efficiently during long-horizon agentic tasks remains challenging for powerful large-scale LLMs. Code and dataset are available at https://github.com/HenryPengZou/InterruptBench.
1 Introduction
InterruptBench addresses a gap in evaluating agents that must adapt to user intent changes during long-horizon, environmentally grounded web tasks. The study formalizes interruption scenarios, builds a benchmark and simulation framework, and evaluates adaptation across models and interaction settings.
- Long-horizon agents must handle hundreds of environmental interactions while remaining interruptible when users correct errors, inspect progress, or revise goals.
- Existing benchmarks assume uninterrupted execution, while interruption studies focus on short, unconstrained language tasks or single-turn settings.
- The paper formalizes addition, revision, and retraction interruptions for web navigation actions that produce persistent state changes.
- InterruptBench synthesizes interruption scenarios from WebArena-Lite under semantic consistency and necessity constraints for reliable evaluation.
- Experiments evaluate six LLM backbones across three interruption scenarios, measuring both adaptation effectiveness and efficiency in single- and multi-turn settings.
2 InterruptBench: Benchmark Construction and Simulation
InterruptBench augments WebArena-Lite with human-reviewed, semantically constrained interruption scenarios and replays trajectories with interruptions inserted at progress-aware points. It covers addition, revision, retraction, no-interruption baselines, and both single- and multi-turn interactions.
- Underlying task suite: InterruptBench augments WebArena-Lite, a long-horizon web-task suite with environmental constraints, programmatic checking, five domains, and 165 human-verified tasks.
- Interruption scenarios: Addition expands an incomplete query, revision corrects an erroneous query, and retraction removes previously specified details.
- Interruption scenarios: The benchmark includes a no-interruption baseline in which agents receive only the initial query.
- Data synthesis: Synthetic scenarios preserve the original meaning and ground-truth answer, require every interruption, and use natural user-facing language.
- Interruption simulation: Simulation generates a full initial trajectory, replays it, inserts an interruption at a relative progress point, and continues execution under the updated intent.
- Interruption simulation: The evaluation supports single-turn interruptions and multi-turn interactions with repeated iterative feedback.
3 Experimental Setup
The experiments evaluate six LLM backbones in a common agent scaffold under varied interruption sequences and progress positions. They measure final success, post-interruption adaptation over action steps, efficiency costs, and performance as interruptions accumulate.
- Models and settings: Six LLM backbones are evaluated with identical prompting, tool/API surfaces, termination conditions, and maximum action budgets unless otherwise noted.
- Interruption configurations: Experiments cover addition, revision, and retraction in both single-interruption and multi-interruption episodes, including varying interruption-type sequences.
- Timing strategy: Interruptions occur at 60% of the matched no-interruption trajectory by default, and post-interruption behavior is aligned by action index k.
- Metrics: Success is evaluated against the final post-update intent using String Match, URL Match, and Program Execution metrics.
- Metrics: The post-interruption success curve SR(k) tracks how success changes with the number of post-interruption actions until completion.
- Metrics: Efficiency is measured through output tokens and action count against matched no-interruption runs on the same tasks.
- Metrics: Success, SR(k), and failure patterns are analyzed as the number of interruptions increases.
4 Experimental Results and Analysis
The experiments evaluate how agents adapt to interruptions, the efficiency cost of adaptation, and performance under repeated updates. Results show rapid but model-dependent success gains, substantial recovery differences, and costly failures when updates are poorly integrated.
- Adaptation Dynamics After Interruption: Success rate rises rapidly until approximately k ≈10 and then plateaus, indicating that most adaptation occurs within the first few post-update actions.The early-improvement rate and eventual ceiling vary across models.
- Adaptation Dynamics After Interruption: Revision depends strongly on update integration, while Retraction has stronger no-update baselines and benefits mainly through earlier completion under tight budgets.For Revision, most baselines remain near zero and Opus exceeds 50% by k ≈15; Addition falls between these patterns.
- Efficiency of Adaptation: Token overhead dominates adaptation cost: under Addition, average action overhead ranges from −0.99 to +0.82, while token overhead reaches +1699.1.These values span Opus and Sonnet for action overhead and Haiku for token overhead.
- Efficiency of Adaptation: F/F episodes incur the highest cost inflation, with prolonged failures reaching +2624 tokens for Haiku, +1128 for Sonnet, and +351 for Opus.The pattern is consistent with prolonged but unproductive re-planning.
- Efficiency of Adaptation: F/S recoveries vary by model: Opus has 70 and Sonnet 55, while DeepSeek, Qwen, and Mistral have 35, 32, and 27, respectively.Haiku requires +6.67 actions and +1808 tokens in recovered episodes, whereas Opus shows −2.21 actions and −92.7 tokens.
- Performance Under Multiple Informational Interruptions: Additional interruptions generally improve success, with Claude models improving consistently and Opus reaching 41.82% at n = 3, while open-weight gains are smaller and less stable.DeepSeek-V3.1 gains from n = 1 to n = 2 but slightly regresses at n = 3.
5 Case Study
The case study contrasts treating an interruption as a superficial message with treating it as a state-changing update that requires revising the active web task state.
- The Addition case begins with route comparison to CMU, then adds Randyland as a new origin constraint mid-execution.
- The failure trace acknowledges the interruption but retains the old origin in the “From” field and computed route.
- The correct trace detects intent drift, retracts the previous origin assumption, and edits the active task state before continuing.
6 Conclusion
The paper studies interruption handling in persistent web environments and concludes that capable agents still struggle to reconcile updated intent with environment state. It frames interruptibility as a realistic human–agent collaboration setting requiring evaluation of success, efficiency, and robustness.
- The study introduces controlled Addition, Revision, and Retraction scenarios for long-horizon web navigation with persistent state changes.
- Across six backbones, interruption handling remains a persistent bottleneck, with agents often continuing from stale assumptions or producing answers inconsistent with updated goals.
- Prior human–agent collaboration studies mostly use language-only or abstract communication and fixed tasks rather than persistent environments with changing user intent.
- The benchmark jointly evaluates success, efficiency, and robustness under environment-bound interruptions.
B Web Environment & Dataset Statistics
WebArena-Lite is a curated WebArena evaluation suite with human-verified tasks, rule-based ground-truth trajectories, and synthesized interruption data for three scenarios.
- WebArena contains 812 real-world tasks across Reddit, GitLab, CMS, Map, and OneStopShop environments.
- WebArena-Lite selects 165 representative tasks for human-verified evaluation and uses 647 remaining tasks for training.
- Rule-based Playwright solvers generate ground-truth trajectories for task prototypes across the websites.
- For each of the 165 evaluation tasks, the authors synthesize Addition, Revision, and Retraction interruption data.
C Efficiency Analysis under Revision Interruptions
Revision interruptions frequently produce failures despite successful matched baselines, while efficiency effects vary across models and can trade off interaction length against generation cost.
- F/S disagreements, where the interrupted run fails and the baseline succeeds, dominate S/F disagreements in the revision analysis.Claude-Opus records F/S=86 versus S/F=1, while Claude-Sonnet records F/S=62 versus S/F=1.
- Interruptions reduce average action counts for several models in the reported revision scenario.
- Table D.2 organizes retraction efficiency statistics by paired outcomes between interrupted and no-interruption runs.
- The efficiency impact of interruption handling is highly model- and scenario-dependent.The comparison can trade off interaction length against generation cost.
D Efficiency Analysis under Retraction Interruptions
Under retraction interruptions, agents can recover some baseline failures while generally using fewer actions and, for most models, fewer tokens.
- Interruptions shift more outcomes to success/success and include non-trivial success/failure cases, suggesting correction of some baseline failures alongside new failures.The reported pattern includes both corrected failures and introduced failures.
- −1.72 to −8.13 Avg. for number of action indicates that all models use fewer actions with interruptions.The negative averages are consistent with shorter action sequences after user updates are incorporated.
- Token usage decreases for most models when interruptions are incorporated.
E Evaluation on Different Interruption Position
Interruption timing affects adaptation differently by model scale: large-scale models benefit from later updates, whereas smaller models become more vulnerable to delayed interruptions.
- Large-scale models achieve their best overall success when interruptions occur later in the baseline trajectory.This pattern indicates effective reuse of progress made before the interruption and more localized trajectory changes.
- Interruption positions of 0.2, 0.4, 0.6, and 0.8 probe how committed intermediate state affects post-update adaptation.
- Relatively smaller models degrade as interruptions are delayed, showing greater sensitivity to stale or noisy intermediate state.
F Prompt for Data Synthesis
The data-synthesis prompts generate realistic addition, revision, retraction, and mixed multi-turn interruption scenarios while preserving the original intent and ground-truth answers.
- F Prompt for Data Synthesis: Four synthesis strategies model additions, revisions, retractions, and mixed multi-turn interactions during web navigation.Claude-Opus-4.5 generates the synthetic variations from WebArena tasks.
- F.1 Addition Strategy: Addition scenarios transform complete specifications into incomplete initial queries followed by updates that add missing details.The strategy simulates users who begin vaguely and progressively clarify requirements.
- F.1 Addition Strategy: Addition prompts require the transformed query plus updates to preserve the original meaning and answers, with every update essential to the ground truth.Updates are concise and written in a realistic user or customer tone.
- F.2 Revision Strategy: Revision scenarios introduce plausible incorrect details, such as dates, products, quantities, or locations, then generate corrections that recover the original intent.Each correction must be essential for reproducing the original ground-truth answers.
- F.3 Retraction Strategy: Retraction scenarios add extra filters or constraints initially, then generate updates that remove them to recover the original intent.The added constraints and retractions must be plausible, natural, and individually essential to the resulting answers.
- F.4 Multi-Turn Mixed Strategy: Mixed multi-turn scenarios combine missing, incorrect, and extra details in an initial query, followed by randomized additions, modifications, and retractions.The updates are labeled by type and must collectively preserve the original meaning and ground-truth answers.
- F.4 Multi-Turn Mixed Strategy: A route-comparison example removes an incorrect starting location so the query recovers the intended route from Randyland to Carnegie Mellon University.