PaperlayerView exact revision on arXiv

Source-linked AI summary

Async-RBench: Benchmarking Asynchronous Task Execution in LLM Agents

Detao Zeng, Guilin Ou, Nuo Chen, Yunhao Feng, Hongliang Wang, Qinli Yang, Junming Shao, Zhong Zhang

arXiv:2610.32429v1cs.AI

TL;DR

Subagent results can arrive after plans, artifacts, or validation have changed, so agents must distinguish work to update from valid work to preserve. Async-RBench evaluates those responses using controlled delivery and event-specific contracts; across nine agents, asynchronous delivery had lower task-success point estimates than batched delivery.

  • Problem

    Subagent results can arrive after plans, artifacts, or validation change, requiring updates to invalidated work without repeating side effects on unaffected work.

    Source: §1 · pp. 1–2

  • Method

    ASYNC-RBENCH uses controlled release conditions and per-event contracts to score response quality separately from task success across 200 executable tasks.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    All nine agents had lower task-success point estimates under asynchronous than batched delivery (26.0% vs. 29.7% on average), and DRS reached at most 45.1.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The findings motivate tracking which work a result affects and which remains valid, evaluated through both event responses and end-to-end task success.

    Source: §5 · p. 9

    Each task isolates one target event, and controller-based delivery with timeout fallback may not reflect real arrivals.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large language model (LLM) agents increasingly execute tasks asynchronously: a main agent continues working while its subagents run, and the task state may change before their results arrive. The main agent must decide which work to update or preserve and which checks to repeat, while avoiding prohibited changes. Final task success alone does not reveal whether these results were handled correctly, and natural asynchronous arrivals are hard to reproduce. We introduce Async-RBench, a benchmark for asynchronous task execution in LLM agents, with 200 executable tasks covering eight scenarios. A scenario controller regulates result delivery using predefined task-state conditions, while an Event Response Contract (ERC) specifies what each response must change, preserve, avoid, and re-verify. The Dynamic Replanning Score (DRS) weights the response process and outcome equally and is reported separately from task success. We evaluate nine agents and find DRS scores of 19.2 to 45.1, with the lowest scores on duplicate reports and subagent failures. Under our execution protocol, all nine agents obtain lower task success with asynchronous than with batched delivery (26.0% vs. 29.7% on average). For three agents, prompting the main agent to rebuild its plan after every result improves DRS by only 2.6 to 3.5 points. Together, these findings highlight the challenges of reliable asynchronous task execution and the need to evaluate how agents handle incoming results alongside end-to-end task success.

1 INTRODUCTION

Async-RBench evaluates how agents handle subagent results that arrive after task state has changed, separating event response from end-to-end task success. Its evaluation finds that asynchronous execution remains challenging across agents.

1 INTRODUCTION

  • A result may invalidate prior work, require preserving unaffected progress, or call for rerunning checks; ignoring it or rebuilding unaffected work can both waste task resources.In the repository example, cleanup after passing tests means the earlier report no longer verifies the code base.Source: §1 · pp. 1–2
  • Existing benchmarks do not isolate responses to subagent results whose validity, recency, or scope must be assessed, and natural asynchronous arrivals are difficult to reproduce.Final task success alone can miss incorrect event handling, including repeated side effects.Source: §1 · p. 2
  • Async-RBench uses 200 executable tasks across eight interruption scenarios, controlled result release, and evaluator-only contracts that define what to change, preserve, avoid, and validate.The Dynamic Replanning Score weights response process and event outcome equally and is reported separately from task success.Source: §1 · p. 2
  • DRS ranges from 19.2 to 45.1, with the lowest scenario means on Duplication and replay (8.6) and Subagent failure (12.2).Under the execution protocol, all nine agents have lower task success estimates with asynchronous than with batched delivery (26.0% vs. 29.7% on average).Source: §1 · p. 2

2 RELATED WORK

Prior agent benchmarks cover task outcomes, collaboration, progress, state integrity, and process quality. Async-RBench focuses on scoring a main agent’s response to each asynchronous subagent-result event separately from task success.

2 RELATED WORK

  • Unlike post-hoc failure diagnosis or general process checks, Async-RBench scores each target-event response against an event-specific contract and reports DRS separately from task success.The closest benchmark, ClawArena-Team, evaluates delegation and integration but does not define a separate response contract for each subagent-result event.Source: §2 · p. 3
  • Related work studies parallel tool execution, orchestration and replanning, evolving environments, interruptions, and delegation failures, but a subagent result may be stale, duplicated, partial, or authoritative.The correct response may therefore be to leave existing work unchanged.Source: §2 · p. 3
  • The benchmark overview depicts concurrent subagents, controlled result release, task-level success checks, and separate ERC-based evaluation of the target response.Reports from B and C can be held until A’s report is presented.Source: Figure 2 · p. 4

3 ASYNC-RBENCH

Async-RBench controls when subagent results reach a working main agent and evaluates each target response against an event-specific contract. It covers 200 tasks grouped across eight scenarios.

3 ASYNC-RBENCH

  • Controlled delivery and per-event contracts address the evaluation obstacles by making release conditions predefined and scoring event response separately from task success.Source: §3 · p. 4

3.1 CONTROLLED ASYNCHRONOUS DELIVERY

  • The controller buffers results until observable task-state prerequisites are met, presents at most one new result per main-agent request, and leaves the agent to judge validity, scope, and needed revisions.A handling window follows each result; a maximum-hold fallback prevents indefinite blocking.Source: §3.1 · p. 4

3.2 EVENT RESPONSE CONTRACTS AND SCORING

  • The ERC scores required changes, valid work to preserve, prohibited changes, required validation, and the event outcome, distinguishing process quality from final-state success.A later repair can improve the outcome score but does not erase credit lost for a prohibited side effect.Source: §3.2 · p. 5
  • DRS averages process scores across applicable contract components and combines process and outcome scoring; task success is the percentage of runs meeting all task goals.Overall DRS is the unweighted mean of the eight scenario scores.Source: §3.2 · p. 5

3.3 TASKS AND SCENARIOS

  • The evaluation set contains 200 tasks adapted from five benchmarks, with 40 tasks per source and 25 per scenario across eight scenarios.Each task has one primary scenario label, while its ERC determines what must be revised or retained.Source: §3.3 · p. 5, Table 2 · p. 6

4 EXPERIMENTS

The experiments evaluate nine agents’ responses to asynchronous subagent results, validate the Dynamic Replanning Score (DRS), and compare task success under asynchronous and batched delivery. Agents score low on DRS overall, and all nine have lower task-success estimates with asynchronous delivery under the tested protocol.

4.1 EXPERIMENTAL SETUP

  • The setup evaluates nine agents with asynchronous results presented individually and batched results presented after required subagent workflows terminate.Runs are paired by task and repetition index; each agent completes 600 runs per delivery mode.Source: §4.1 · p. 6

4.3 AGENT PERFORMANCE UNDER ASYNCHRONOUS DELIVERY

  • DRS ranges from 19.2 for Gemini 3 Flash High to 45.1 for Claude Sonnet 5, while Oracle scores range from 96.4 to 98.8.Even the highest agent DRS is below 50 on the 0–100 scale.Source: Table 3 · p. 7, §4.3 · p. 7, Figure 3 · p. 7

4.2 VALIDATION OF DRS

  • DRS separates response conditions: Frozen scores 3.7–16.1, Native 19.2–39.7, and Oracle 96.4–98.8 across three agents.Oracle is a requirement-aware reference, not a capability-matched baseline.Source: §4.2 · p. 7, Figure 3 · p. 7

4.3 AGENT PERFORMANCE UNDER ASYNCHRONOUS DELIVERY

  • Mean DRS is lowest for Duplication and replay (8.6) and Subagent failure (12.2), versus 57.5 for Scope and dependencies and 60.7 for Resource stragglers.The low-scoring scenarios require preserving valid work and avoiding unnecessary side effects; scenario gaps alone do not establish why those cases are harder.Source: §4.3 · pp. 7–8
  • DRS rankings vary by scenario and differ from task-success rankings; more model calls also do not necessarily correspond to higher DRS.Terra scores 39.7 with 48.1 calls per run, while Kimi K2.5 scores 26.1 with 125.2 calls.Source: §4.3 · p. 8

4.4 TASK SUCCESS IS LOWER UNDER ASYNCHRONOUS DELIVERY

  • All nine agents have lower task-success estimates with asynchronous than batched delivery, averaging 26.0% versus 29.7%.Eight agents also consume more tokens under asynchronous delivery.Source: Table 4 · p. 8, §4.4 · p. 8

4.5 MANDATORY PLAN REBUILDING YIELDS LIMITED GAINS

  • Full-rebuild raises DRS for Terra (39.7 to 42.3), Luna (31.1 to 34.6), and Gemini (19.2 to 21.9), while mean calls rise by 7.5–10.2%.The study does not report whether agents followed the rebuild instruction, so it cannot distinguish noncompliance from errors that persist after rebuilding.Source: Figure 6 · p. 9, §4.5 · p. 9

5 CONCLUSION

Async-RBench evaluates how LLM agents handle arriving subagent results, and nine agents show low DRS, lower task success under asynchronous delivery, and limited gains from rebuilding plans after each result. The authors identify scope and measurement constraints in the benchmark.

5 CONCLUSION

  • DRS reaches at most 45.1 across nine agents and is lowest on duplicate reports and subagent failures.Under the execution protocol, all nine agents have lower task-success point estimates with asynchronous than batched delivery.Source: §5 · p. 9
  • Rebuilding the plan after every result improves DRS by only 2.6–3.5 points for three agents, while increasing model calls by 7.5–10.2%.Source: §5 · p. 9
  • The benchmark isolates one target event per task, and its controller-based delivery with timeout fallback may not reflect real arrivals.DRS also scores valid runs that terminate before target delivery as zero; shared models, independent-run timing variation, and human-written ERCs and verifiers constrain interpretation.Source: §5 · p. 9

AI USE STATEMENT

The authors used generative AI for research-support tasks and writing and presentation work, then reviewed the assisted work and take responsibility for the final paper.

AI USE STATEMENT

  • Generative AI produced initial annotation labels, distilled open-source documentation into the skill library, and assisted with framework and analysis-code implementation.Source: AI USE STATEMENT · p. 10
  • The authors also used generative AI to draft and polish text, create figures, and format references.Source: AI USE STATEMENT · p. 10
  • The authors manually audited all labels, verified and tested all code, and accept responsibility for the final content.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

The paper describes versioned task construction, execution acceptance, and run-level records intended to make benchmark materials and analyzed runs traceable.

REPRODUCIBILITY STATEMENT

ETHICS STATEMENT

The paper reports no human subjects, sensitive personal data, or potentially harmful applications, and describes task sourcing, human-reviewed construction, and evaluation-set composition.

ETHICS STATEMENT

  • The study involves no human subjects, sensitive personal data, or potentially harmful applications; source tasks come from published benchmarks and follow their usage licenses.Source: ETHICS STATEMENT · p. 10

A.1 TASK SOURCES AND CONSTRUCTION PROCESS

  • Researchers use source-task evidence to design delegated workflows, result versions, release conditions, and ERCs, with human review before executable acceptance.Figure 7 summarizes screening, human-led scenario and ERC design and review, and execution checks in both delivery modes.Source: §A.1 · p. 15, Figure 7 · p. 15

A.2 TASK COMPOSITION AND AGGREGATION

  • The evaluation set contains 200 tasks across eight equally sized scenarios and five sources, with 25 tasks per scenario and 40 per source.Tasks are weighted equally, so each scenario contributes one eighth of the overall result under complete coverage; the 47-task construction set is excluded.Source: §A.2 · p. 15

A.3 SAMPLED HUMAN REVIEW AND VERIFIER ACCEPTANCE

The sampled audit found 96.6% agreement between human and automated verifier judgments, with disagreements traced to infrastructure failures. The taxonomy and case specifications define event-specific response requirements while leaving systems to choose their action sequence.

A.3 SAMPLED HUMAN REVIEW AND VERIFIER ACCEPTANCE

  • 96.6% item-level agreement was found between human judgments and automated verifier decisions in the initial audit of 405 sampled runs.All disagreements traced to external infrastructure failures; reruns replaced affected records, but the conclusions apply only to the audited sample and versions.Source: §A.3 · pp. 15–16

A.4 ASYNCHRONOUS SCENARIO TAXONOMY

  • Eight scenarios organize result events by validity, information completeness, task dependencies, and execution scheduling, with task evidence determining specific response requirements.Responses may require revision, preservation, avoiding forbidden changes, and revalidation.Source: §A.4 · p. 16

A.5 CASE SPECIFICATION AND EXECUTION ACCEPTANCE

  • The ERC binds required changes, preservation, forbidden changes, revalidation, and outcomes to concrete artifacts or evidence without prescribing a unique action sequence.Execution acceptance checks event triggers, submission conditions, verifier behavior, feasible completion paths, and detection of invalid responses.Source: §A.5 · p. 16

B REPRESENTATIVE CASES ACROSS EIGHT SCENARIOS

  • The appendix presents one recorded run for each scenario, using excerpts of task context, subagent results, and subsequent main-agent actions.The records shown do not independently establish task success.Source: B REPRESENTATIVE CASES ACROSS EIGHT… · p. 17

B.1 DELAYED AUTHORITY: SELECT A DEPLOYMENT BACKEND

  • In delayed authority, the main agent uses authoritative profile v2 for deployment configuration and batch lineage despite a later candidate report citing v1.Arrival time does not necessarily identify the more current basis for deployment.Source: §B.1 · p. 17

B.2 STALENESS AND ORDERING: FIX BOTTLE HEADER INJECTION

  • In the header-injection case, a later patch report relies on pre-rewrite revisions, so its test results cannot substitute for checks of the current repository.The main agent runs a local header check that exits successfully after the report arrives.Source: §B.2 · p. 17

B.3 CONFLICTING RESULTS: RECOVER AND REMOVE A GIT SECRET

The recorded cases show that asynchronous responses require checking report scope, identity, completeness, and relevance to current work. Execution rules score response process separately from final task success and specify how runs are included.

B.3 CONFLICTING RESULTS: RECOVER AND REMOVE A GIT SECRET

  • In the Git-secret case, scans differed because they covered different repository scopes; the main agent used the unreachable-object finding for recovery.The cleanup report records reflog expiration and garbage collection.Source: §B.3 · pp. 17–18

B.4 DUPLICATION AND REPLAY: BUILD OFFICETASKSCHEDULER

  • The OfficeTaskScheduler trace shows a replayed completion was accepted again after the main agent had already changed its decision, but does not establish duplicate execution.Assessing the response requires checking completion identity, the existing decision, and actual side effects separately.Source: §B.4 · p. 18

B.5 PARTIAL TO COMPLETE: FIX SYMPY DIMENSION INFERENCE

  • In the SymPy case, fuller reports expanded the repair from an exponential-function special case to generic function handling, then limited normalization for user-defined functions.The trace records successive changes to the repair’s scope.Source: §B.5 · p. 18

B.6 SUBAGENT FAILURE: BUILD THE MULTISERVE DELIVERY APP

  • In the MultiServe case, the main agent rejects an unusable delivery implementation while accepting work from three other subagents and implementing the logistics logic itself.The subagent result failed to match the actual restaurant, order, and courier interfaces; this was not a crashed process.Source: §B.6 · pp. 18–19

B.7 SCOPE AND DEPENDENCIES: FIX PYLINT REGEX PARSING

  • The Pylint case requires handling commas in regex quantifiers without breaking ordinary list parsing or configuration-loading paths.Reports supply test cases for those other callers.Source: §B.7 · p. 19

B.8 RESOURCE STRAGGLERS: FIX MATPLOTLIB FIGURE PICKLING

  • In the Matplotlib case, existing pickling tests pass but omit figures after align_labels(), while a targeted check finds unpicklable weak-reference objects.The main agent changes how the relevant grouping object saves and restores its state.Source: §B.8 · p. 19

C EXECUTION DETAILS AND COST ANALYSIS

  • Async delivery presents at most one new result per main-agent call, while scoring runs from presentation through termination; task success and ERC process compliance are separate.Early termination leaves prescribed unpresented events scored under the applicable DRS rule; runs are paired by task and repetition, with reruns replacing invalid attempts rather than selecting the best score.Source: §C.1 · p. 21, §C.2 · p. 21, §C.4 · p. 22

C.3 WORKED DRS EXAMPLE

  • 83.3 was the worked example’s DRS, combining a process score of 2/3 with a final outcome score of 1.The proposal added throughput and peak-memory evaluation while retaining reconstruction and generation metrics, but no substantive post-event validation was recorded.Source: §C.3 · p. 22, Table 6 · p. 22

C.5 COST COMPOSITION AND RESPONSE QUALITY

Asynchronous delivery raises token consumption for most agents, with much of the increase occurring in the main agent. Similar execution costs can accompany different DRS scores.

C.5 COST COMPOSITION AND RESPONSE QUALITY

  • Eight of nine agents consume more tokens on average under asynchronous delivery; Gemini 3 Flash High decreases by 2.3%.The increases span the distribution rather than being confined to a few costly cases; Qwen3.5 27B’s median is nearly unchanged while its 90th percentile rises from 2.67M to 2.97M tokens.Source: §C.5 · p. 23, Figure 12 · p. 23
  • Mean main-agent token consumption rises for all nine agents, and the main agent’s share of total consumption increases for every agent.For Terra, main-agent consumption increases by 100.2% while subagent consumption is nearly unchanged; Figure 5 shows the component comparison.Source: §C.5 · p. 23
  • Sonnet and DeepSeek consume 1.67M and 1.66M tokens on average, yet their DRS values are 45.1 and 30.6, respectively.The comparison is across agents and does not show how changing one agent’s budget affects its score; Figure 13 plots DRS against mean execution cost.Source: §C.5 · p. 23, Figure 13 · p. 24
  • The cost and response-quality comparisons motivate controlled tests of response frequency, context management, and revision strategy while holding models and tasks fixed.The proposed comparison would test whether execution cost can be reduced while maintaining event response quality.Source: §C.5 · p. 23

C.6 EXECUTION AND SCORING VERSIONS

  • Reproduction binds the task set, model and resource configurations, case and verifier versions, execution framework, and individual scoring decisions.The manuscript uses the v11.0 execution protocol and Section 3.2 scoring rules.Source: §C.6 · p. 24

D FAILURE CATEGORIES AND TRAJECTORY EXCERPTS

The trajectory excerpts document several response and evidence-handling issues, while paired runs show different outcomes under asynchronous and batched delivery. These examples do not establish that interruption caused the errors.

D FAILURE CATEGORIES AND TRAJECTORY EXCERPTS

  • Async traces include unreliable verification, execution-state confusion, goal drift, evidence distortion, and missed global constraints, though some issues do not cause final failure.The rejected import is later corrected, and the flawed test’s failure branch is not shown to have been reached.Source: D FAILURE CATEGORIES AND TRAJECTORY… · p. 24, Figure 14 · p. 25
  • Two WAL recovery runs succeed under batched delivery but fail under Async after acting on assumptions before examining the encrypted log; paired batched runs decode and replay it.Neither mode’s subagent reports supply the full answer, and the Async outputs fail data-integrity and log-update checks.Source: D FAILURE CATEGORIES AND TRAJECTORY… · p. 24
  • Figure 14 presents one trajectory excerpt per issue category, with event order marking result presentation and subsequent main-agent actions.Temporal order alone does not show that interruption caused an error.Source: Figure 14 · p. 25
Loading 2610.32429v1…