PaperlayerView exact revision on arXiv

Source-linked AI summary

DynaWeave: Learning to Adapt through Dynamic Scenario Synthesis

Xuan Yang, Chunxiao Liu, Hongsheng Xin, Hao Xu, Ning Miao

arXiv:2610.29443v1cs.CL

TL;DR

Many tool-use environments remain static during task execution, although external events can invalidate plans and require agents to adapt. DynaWeave synthesizes verifiable dynamic scenarios from static tasks and uses verified trajectories to fine-tune Qwen3.5 models. Training improves performance across Gaia2, CostBench, and ToolMaze, including a 67.34-percentage-point CostBench completion gain for the 2B model.

  • Problem

    Many environments assume stability after a task begins, despite external changes that can invalidate plans, while constructing dynamic scenarios and verifiers at training scale requires substantial manual effort.

    Source: §1 · pp. 1–2

  • Method

    DynaWeave generates task-relevant exogenous events, activates them during interaction, and adapts only verifier checks affected by events that occur, retaining unaffected requirements.

    Source: §1 · p. 2

  • Results

    Across Gaia2, CostBench, and ToolMaze, training improves overall performance across all three Qwen3.5 sizes, with 2B CostBench task completion increasing by 67.34 percentage points.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    Training on verified dynamic-scenario trajectories improves Qwen3.5 performance across the evaluated benchmarks, supporting generalization beyond the training environments.

    Source: §5 · p. 9

    Models still struggle with incorrect tool results lacking explicit errors and with responding to changes at the right time, potentially because training scenarios lacked these changes.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Large language model (LLM) agents increasingly use tools to complete tasks in stateful environments. However, many training environments remain largely static apart from changes caused by the agent's own actions, whereas external events introduced by other users, services, or sudden contingencies in real environments can invalidate ongoing plans. To address this gap, we introduce DynaWeave, an automated synthesis framework that transforms existing static tasks into verifiable dynamic scenarios. The framework generates exogenous events relevant to each task, covering changes in task requirements, environment state, and external information. It uses directed acyclic graphs (DAGs) to represent dependencies between events and an execution engine to activate them during agent interaction. A dynamic verifier revises only the evaluation checks affected by events that actually occur and retains all other checks. Using DynaWeave, we collect 2,095 verified interaction trajectories across 36 environments and use them for supervised fine-tuning of Qwen3.5 models. Experiments on Gaia2, CostBench, and ToolMaze show that DynaWeave achieves consistent improvements in overall performance across all three evaluated model sizes, increasing the task completion rate on CostBench by 67.34 percentage points and even exceeding the performance of Gemini-3.1-Pro-Preview on ToolMaze. These results support dynamic scenario synthesis as a practical approach to training agents to adapt to changing environments. Our code is available at https://github.com/Miaow-Lab/DynaWeave.

1 INTRODUCTION

Existing tool-use environments largely assume stability during a task, although external changes can invalidate plans and require agents to revise decisions. DynaWeave synthesizes verifiable dynamic scenarios from static tasks; training on its trajectories improves Qwen3.5 performance across three benchmarks.

1 INTRODUCTION

  • Most environments assume that only the agent changes the environment, but external events can invalidate otherwise correct plans and make success depend on revising decisions.Examples include another user claiming a resource, a service becoming unavailable, or an unexpected contingency.Source: §1 · p. 1
  • Constructing dynamic scenarios and their verifiers requires substantial manual effort, limiting production at the scale needed for agent training.Recent dynamic benchmarks evaluate adaptation, but their construction does not remove this scaling challenge.Source: §1 · pp. 1–2
  • DynaWeave generates and verifies exogenous events for existing tasks, activates them during interaction, and adapts affected checks while preserving unaffected requirements.It filters invalid, implausible, or task-irrelevant events and supports extending existing task collections without manually designing dynamic scenarios for each domain.Source: §1 · p. 2
  • Across Gaia2, CostBench, and ToolMaze, training consistently improves performance across three Qwen3.5 sizes; 2B CostBench task completion rises by 67.34 percentage points.The 9B model also reaches a ToolMaze composite score of 32.28, exceeding Gemini-3.1-Pro-Preview’s 32.19.Source: §1 · p. 2
  • The work contributes a framework, 2,095 dynamic scenarios across 36 environments, and verified trajectories used to train agents to adapt to changing conditions.The scenarios are derived from EnvScaler, τ 2-bench, ToolSandbox, and AgentDojo.Source: §1 · p. 2

2 RELATED WORK

Prior work supplies executable and dynamic benchmarks, while DynaWeave complements these efforts by deriving plausible within-task changes and verification rules from existing static tasks. Its scope preserves the source environment and tool interface.

2.1 EXECUTABLE BENCHMARKS AND ENVIRONMENT SYNTHESIS

  • Executable benchmarks provide reusable applications and evaluation procedures, while environment-synthesis systems expand the supply and diversity of executable tasks.Examples include WebArena and OSWorld for web and computer systems, and EnvScaler for environment skeletons, task scenarios, and validators.Source: §2.1 · p. 2
  • DynaWeave complements environment synthesis by generating changes within existing tasks, specifying their observability, and adapting evaluation while reusing the source environment and tools.Separating event application from agent behavior allows scenario-generation errors to be checked independently of adaptation failures.Source: §2.1 · p. 3

2.2 DYNAMIC AND ASYNCHRONOUS AGENT BENCHMARKS

  • Dynamic benchmarks cover distinct adaptation challenges, including asynchronous updates, replanning under changing costs, tool failures, and changing evidence.The settings also include multi-day workflows, temporal changes, and recovery from disrupted execution.Source: §2.2 · p. 3
  • Unlike approaches that evolve environments across versions or tasks, DynaWeave retains tools and environment rules while introducing changes during task execution.This supports paired evaluation of static and dynamic variants and demonstrations of adaptation in the same underlying environments.Source: §2.2 · p. 3

3 DYNAWEAVE

DynaWeave turns executable static tasks into dynamic scenarios by generating events, controlling their activation, and revising verification only for changes that occur. It preserves the source interaction interface and evaluates task requirements against the realized execution.

3 DYNAWEAVE

  • DynaWeave transforms static tasks into verifiable dynamic scenarios using event construction, controlled execution, and verifier adaptation.The verifier can also check execution traces when correctness depends on action timing or order.Source: §3 · p. 3

3.1 EXOGENOUS EVENT CONSTRUCTION

  • Event construction identifies task-relevant change sources, specifies each event’s effects, trigger conditions, and observability, then organizes compatible events in a validated DAG.Themes include revised task requirements, changed environment state, and external information; observability may be loud, silent, or partial.Source: §3.1 · pp. 3–4, Figure 1 · p. 4

3.2 CONTROLLED DYNAMIC EXECUTION

  • Events activate after specified actions, at reproducible virtual-time boundaries, or randomly under a fixed seed; each event occurs at most once and only realized changes are recorded.Action-triggered events occur after the tool call finishes, while virtual-time events activate at the first eligible interaction boundary at or after their scheduled time.Source: §3.2 · pp. 4–5

3.3 DYNAMIC VERIFIER ADAPTATION

  • The dynamic verifier revises checks affected by realized events, retains unaffected requirements, and removes a task requirement only when the user explicitly cancels it.It validates proposed modifications and checks agreement with the original verifier when events are disabled; strict validation tests both invalidated static and adapted solutions.Source: §3.3 · p. 5

3.4 INTERACTION WITH DYNAMIC SCENARIOS

  • Dynamic interaction keeps the source tools and action space while extending hidden state with environment information, current requirements, and event-execution records.Observations include tool results and permitted notifications; teacher trajectories record these observations and the actions taken in response to changes.Source: §3.4 · p. 6

4 EXPERIMENTS

DynaWeave fine-tuning improves performance across Gaia2, CostBench, and ToolMaze, with gains spanning model sizes and dynamic conditions. The experiments also show diminishing returns from more trajectories and complementary value from diverse event sources.

4.1 EXPERIMENT SETUP

  • The experiments evaluate 2,095 verified trajectories collected across 36 environments and test generalization on Gaia2, CostBench, and ToolMaze.The environments cover asynchronous application changes, changing travel-planning constraints, and explicit or implicit tool failures.Source: §4.1 · pp. 6–7

4.3 DATA EFFICIENCY AND PERFORMANCE SATURATION

  • With 1,048 trajectories, Qwen3.5-4B reaches 87.08% CostBench completion and a 28.91 ToolMaze composite score, versus 90.81% and 29.99 with 2,095 trajectories.Gains diminish with more data, and efficiency metrics do not improve consistently; Figure 3 shows training-scenario quantity effects on CostBench and ToolMaze.Source: §4.3 · pp. 7–8, Figure 3 · p. 8

4.2 MAIN RESULTS

  • Across all three model sizes, DynaWeave improves Gaia2 and ToolMaze overall scores and CostBench task completion; Qwen3.5-4B reaches 5.0 on Gaia2, 90.81% CostBench completion, and 25.38% ToolMaze success under perturbations.The trained 4B model also exceeds the original 9B model on Gaia2 overall and CostBench task completion.Source: §4.2 · p. 7, Table 1 · p. 8

4.4 CONTRIBUTIONS OF DIFFERENT EVENT SOURCES

  • The mixed event-source subset performs best on CostBench completion and ToolMaze composite score, while excluding Task Requirement or Environment State events most harms the respective task and recovery outcomes.A 524-trajectory mixed subset outperforms 1,048-trajectory single-source omissions on several ToolMaze measures, but coverage and optimization-step differences prevent attributing this solely to source diversity.Source: §4.4 · pp. 8–9, Table 2 · p. 9

5 CONCLUSION

DynaWeave converts static executable tasks into verifiable dynamic scenarios and supports supervised fine-tuning with verified interaction trajectories. The conclusion reports improved performance across three benchmarks while identifying limits in efficiency and timely adaptation.

5 CONCLUSION

  • DynaWeave generates exogenous events across task requirements, environment state, and external information, then adapts the original verifier to changes that occur.It preserves original tools and unaffected task requirements.Source: §5 · p. 9
  • Supervised fine-tuning on 2,095 verified trajectories from 36 environments improves all three evaluated Qwen3.5 sizes on Gaia2, CostBench, and ToolMaze.The conclusion presents this as supporting generalization beyond the training environments.Source: §5 · p. 9
  • The conclusion reports diminishing gains with more data, uneven alignment between task completion and efficiency, and continued difficulty responding to changes at the appropriate time.It identifies improving scenario construction and training for these limitations as future work.Source: §5 · p. 9

A DETAILED EVALUATION RESULTS

The ToolMaze detailed results distinguish execution without injected failures from explicit failures and implicit output corruption, each of which may be transient or permanent.

A DETAILED EVALUATION RESULTS

  • ToolMaze conditions include no injected failures, explicit transient or permanent failures, and implicit transient or permanent output corruption.NP denotes no perturbation; P1/P2 are explicit failures, and P3/P4 are implicit corruption conditions.Source: A DETAILED EVALUATION RESULTS · p. 12

Related papers

Loading 2610.29443v1…