PaperlayerView exact revision on arXiv

Source-linked AI summary

STT-Arena: A More Realistic Environment for Tool-Using with Spatio-Temporal Dynamics

Tingfeng Hui, Hao Xu, Pengyu Zhu, Hongsheng Xin, Kun Zhan, Sen Su, Chunxiao Liu, Ning Miao

arXiv:2605.18548v1cs.CLcs.AI

TL;DR

Existing dynamic benchmarks emphasize timely responses to environmental change, while adaptive replanning after spatio-temporal disruptions remains less explored. STT-Arena evaluates replanning in interactive tasks with injected spatio-temporal triggers, and STT-Agent-4B combines iterative trajectory refinement with verifiable dynamic tasks for online RL. The strongest frontier model achieved 35.39%, while STT-Agent-4B achieved 27.17% on STT-Arena, matching GLM-5.1.

  • Problem

    Existing dynamic benchmarks emphasize timely responses to environmental change, while adaptive replanning after spatio-temporal disruptions remains less explored.

    Source: §1 · p. 2

  • Method

    STT-Arena evaluates replanning in interactive tasks with injected spatio-temporal triggers, and STT-Agent-4B combines iterative trajectory refinement with verifiable dynamic tasks for online RL.

    Source: §1 · pp. 2–3

  • Results

    The strongest frontier model achieved 35.39%, while STT-Agent-4B achieved 27.17% on STT-Arena, matching GLM-5.1.

    Source: §1 · p. 3, §5 · p. 10

  • Takeaways & Limitations

    STT-Arena provides a benchmark for evaluating adaptive replanning under spatio-temporal dynamics, and the authors report that targeted training on observed failure patterns can produce competitive performance against many open-source frontier models.

    Source: §1 · p. 2, §5 · p. 10

    The 227 instances and nine conflict subtypes may not cover the full spectrum of real-world spatio-temporal conflict patterns.

    Source: B Limitations and Potential Society… · p. 13

Abstract

from arXiv · show

Large language models (LLMs) deployed in real-world agentic applications must be capable of replanning and adapting when mid-task disruptions invalidate their prior decisions. Existing dynamic benchmarks primarily measure whether LLMs can detect temporal changes in a timely manner, leaving the complementary challenge of adaptive replanning under spatio-temporal dynamics largely unexplored. We introduce STT-Arena (Spatio-Temporal Tool-Use Arena), a benchmark of 227 high-quality interactive tasks spanning nine spatio-temporal conflict types and four solvability levels. Each task is grounded in a realistic, executable environment equipped with injected spatio-temporal triggers that can abruptly invalidate an ongoing plan, forcing the model to detect the state shift and construct a revised execution strategy. Extensive evaluation of frontier LLMs reveals that even the SOTA proprietary models, including Claude-4.6-Opus, achieves less than 40\% overall accuracies, highlighting the fundamental difficulty of spatio-temporal dynamic reasoning. Systematic analysis of failure trajectories uncovers three recurring error modes of existing models: Stale-State Execution, Misdiagnosis of Dynamic Triggers, and Missing Post-Adaptation Verification. Guided by these findings, we propose an iterative trajectory refinement technique that eliminates these failure patterns from training data, and combine it with online RL to produce STT-Agent-4B which outperforms frontier LLMs on STT-Arena.

1 Introduction

Existing dynamic benchmarks emphasize timely response to environmental change, while adaptive replanning after a change invalidates a plan remains less explored. STT-Arena evaluates this capability through executable spatio-temporal tasks and supports targeted training for tool use under changing conditions.

1 Introduction

  • STT-Arena evaluates adaptive replanning and tool use when spatio-temporal changes invalidate prior decisions.The benchmark complements dynamic evaluations focused on timely response to environmental change.Source: §1 · p. 2
  • Its taxonomy organizes authentic tool-use tasks into three categories and nine subcategories, with solvable tasks at Easy, Medium, and Hard levels and a separate impossible level.Each task has an executable environment that simulates spatio-temporal dynamics.Source: §1 · p. 2
  • The simulation infrastructure combines environment curation, spatio-temporal dynamic injection, and dual-agent assessment to support systematic evaluation and targeted improvement.The authors describe it as a reproducible, extensible, high-fidelity testbed.Source: §1 · p. 2
  • The authors analyze recurring failure modes and refine training trajectories by reordering, deleting, or modifying tool-call blocks.The supplied passage begins to describe this approach as building on 2,212 refined trajectories.Source: §1 · p. 2
  • STT-Agent-4B achieves 27.17% on STT-Arena, matching GLM-5.1 after training with refined trajectories and verifiable dynamic tasks for online RL.The paper reports this as the result of its trajectory-refinement and online-RL training approach.Source: §1 · pp. 2–3

2 Spatio-Temporal Tool-Use Arena

STT-Arena models tasks as tool-using interactions in environments where triggers can change state or tool availability and invalidate a plan. Its construction pipeline builds and validates dynamic tasks, while evaluation checks final task success or recognition of infeasibility.

2.1 Task Formulation

  • Each task is a tuple of environment and tools, spatio-temporal triggers, user profile, query, and a checklist of success conditions; triggers can autonomously alter state or tool availability during tool calls.The final environment state is evaluated against the checklist to obtain Pass@1.Source: §2.1 · p. 3

2.2 STT-Arena Construction Pipeline

  • The three-stage construction pipeline curates static environments, injects controlled conflicts to create solvable and impossible tasks across nine types, then uses dual-agent assessment and human review.The final benchmark contains 227 instances.Source: §2.2 · p. 3, §2.2.3 · p. 5

2.2.1 Stage 1: Environment Curation.

  • Stage 1 filters user queries for statefulness and sensitivity to time or location, synthesizes executable environment code, and validates candidate environments through test configurations.Environments that produce runtime errors, empty outputs, or otherwise fail expected tool outputs are discarded.Source: §2.2.1 · p. 4

2.2.2 Stage 2: Spatio-Temporal Dynamic Injection

  • Stage 2 assigns compatible conflict types and difficulty levels, then builds dynamic environments, user queries, profiles, and initial states from blueprints.The blueprints specify triggers, effects, expected post-trigger states, and resolution steps.Source: §2.2.2 · pp. 4–5

2.2.3 Stage 3: Dual-Agent Assessment

  • Stage 3 generates checklists for feasible tasks and verifies that scheduled triggers and effects disrupt the original conflict-free plan as specified.For impossible tasks, the checklist requires recognizing infeasibility and communicating it to the user simulator; LLM auditing and human review check consistency.Source: §2.2.3 · p. 5

2.3 Task Evaluation

  • For feasible tasks, rule-based checks assess the final state against success criteria; for impossible tasks, an LLM judge checks whether the model recognizes and communicates infeasibility.The impossible-task evaluation uses a two-item checklist.Source: §2.3 · p. 5
  • 35.39% is the overall Pass@1 of best-performing Claude-4.6-Opus on STT-Arena.The figure caption groups results into closed-source, open-source, efficient-model, and STT-Agent categories.Source: Figure 3 · p. 6

3 Experiments

The experiments compare closed-source, open-source, and efficient LLMs using Pass@1, with overall scores weighted across solvability categories. They examine benchmark performance, task-specific difficulty, evaluation conditions, and STT-Agent results.

3.2 Main Evaluation: STT-Arena Reveals Fundamental Gaps in Dynamic Reasoning

  • 35.39% overall was Claude-4.6-Opus’s best score; Claude and GPT led closed-source rankings, while Deepseek-V3.2 led open source at 32.16%.The strongest open-source model still trailed the closed-source leaders.Source: §3.2 · pp. 6–7
  • All models’ performance declined from Easy to Hard, with notable drops at Hard, leaving long-horizon replanning under intertwined constraints an open challenge.The authors characterize these results as evidence of the benchmark’s fundamental difficulty.Source: §3.2 · p. 6

3.3 Key Analyses: What Makes STT-Arena Hard?

  • T1 window expiry, S1 site mismatch, and ST1 resource shifts were comparatively manageable; T2 priority reorder, S3 route restriction, ST2 failure cascades, and ST3 handoff failures were difficult.The contrast suggests that spatio-temporal coupling is not inherently difficult; cascading dependencies and misaligned handoffs were especially challenging.Source: §3.3 · p. 7, Figure 4 · p. 7
  • Introducing spatio-temporal triggers reduced performance across all four compared models and substantially changed their relative ordering against static evaluation.The authors conclude that static evaluation alone does not assess robustness to realistic environmental shifts.Source: §3.3 · p. 7, Figure 5 · p. 7

3.4 Further Ablations: Probing Model Capabilities and Design Choices

  • Increasing test-time attempts from one to eight improved all three models, but gains saturated and even the best model remained below 50% at Pass@8.Pass@k counts a task as successful when any of k independent attempts succeeds.Source: §3.4 · p. 8, Figure 6 · p. 8
  • Removing the user simulator consistently reduced performance; without its guidance, models became less confident and often entered local loops or repetitive failures during replanning.The evaluation compared three user simulators with a No User setting.Source: §3.4 · p. 8, Figure 7 · p. 8
  • Thinking mode improved performance for Claude, Deepseek, and GLM, while Qwen showed nearly identical results across modes, possibly because its non-thinking summary supplied structured guidance.The authors caution that reasoning-mode effectiveness depends on how models articulate and integrate their outputs.Source: §3.4 · p. 8, Figure 8 · p. 8

3.5 STT-Agent Results

  • 27.17% overall Pass@1 was achieved by STT-Agent-4B; refined trajectories raised Pass@1 from 23.10% to 25.11% and reduced average API calls from 32.70 to 15.30.Table 2 compares the baseline, training without trajectory refinement, and training with refinement.Source: Table 2 · p. 8, §3.5 · p. 9

4 Discussion

The analysis identifies three recurring failures in dynamic tool use and reports that iterative trajectory refinement targets them while improving STT-Arena performance and reducing tool-call rounds.

4.1 Failure Mode Analysis

  • Stale-state execution occurs when models persist with pre-trigger plans and repeat actions instead of checking the changed environment state.The authors suggest models overcommit to their initial reasoning and underuse new tool observations.Source: §4.1 · p. 9
  • Misdiagnosis occurs when models mistake dynamic triggers for parameter errors, transient glitches, or recoverable obstacles.Successful adaptation requires inferring why the state changed before choosing a response.Source: §4.1 · p. 9
  • Missing post-adaptation verification occurs when models stop after an intermediate action without checking that the updated task constraints are satisfied.Tool success does not ensure that the global state is feasible or the task complete.Source: §4.1 · p. 9

4.2 Iterative Trajectory Refinement

  • Refined trajectories significantly outperform original trajectories, while reducing the average number of tool-call rounds on STT-Arena.The refinement stages target root-cause diagnosis, state refresh, and end-state verification.Source: §4.2 · p. 9

5 Conclusion

STT-Arena evaluates replanning after abrupt spatio-temporal changes, addressing a dimension the authors distinguish from existing tool-use and dynamic benchmarks. Their evaluation finds substantial difficulty for frontier models, while targeted training yields a 4B model competitive with many open-source frontier models.

5 Conclusion

  • Even the strongest frontier model achieves only 35.39% on STT-Arena, while STT-Agent-4B is competitive with many open-source frontier models.The authors combine iterative trajectory refinement with verifiable online reinforcement learning and describe the approach as sample-efficient.Source: §5 · p. 10

A Related Work

  • No-environment benchmarks lack a closed, self-consistent environment, while static-environment benchmarks omit external time- or location-driven state changes.The related-work discussion contrasts API-focused benchmarks with closed environments whose changes are agent-driven.Source: A Related Work · p. 13
  • Recent dynamic benchmarks emphasize gradual or predictable temporal changes rather than abrupt disruptions involving both time and location.STT-Arena allows triggers to change the environment state at any step, forcing detection, replanning, and adaptation.Source: A Related Work · p. 13

B Limitations and Potential Society Impacts

The authors identify limits in benchmark coverage, construction, evaluation, and training scalability, alongside potential benefits and risks of advancing adaptive replanning.

B Limitations and Potential Society Impacts

  • The 227-instance benchmark and its nine conflict subtypes may not cover the full range of real-world spatio-temporal dynamics.The authors also note possible synthesis biases toward patterns or language more readily generated by Qwen-3.5-397B-A17B.Source: B Limitations and Potential Society… · p. 13
  • A fixed passive user simulator cannot fully represent diverse human communication, and refinement overhead and scalability to larger models or multi-agent settings remain underexplored.The authors report that the simulator was ablated across multiple backbone models.Source: B Limitations and Potential Society… · p. 13
  • The authors present improved agent robustness in clinical consultation, logistics, and airline reservation as a potential positive impact.They frame this as an intended implication of developing agents robust to dynamic real-world conditions.Source: B Limitations and Potential Society… · p. 13
  • The authors warn that these advances could lower deployment barriers in sensitive domains before adequate safety guarantees, while synthesis tools could generate misleading adversarial environments.They encourage pairing dynamic-reasoning advances with robustness certification and human oversight.Source: B Limitations and Potential Society… · p. 14
  • All evaluated models struggle most at the Hard level, and performance degrades consistently from Easy to Hard.The table caption also reports STT-Agent-4B at xx% overall; the value is unspecified there.Source: Table 3 · p. 14

C.1 Detailed information of Main Results

The text directs readers to Table 3 for detailed results corresponding to Figure 3.

C.1 Detailed information of Main Results

  • Table 3 reports detailed results corresponding to Figure 3.Source: §C.1 · p. 14

C.2 Detailed Information of STT-Arena

STT-Arena contains 227 manually verified instances spanning solvable and impossible tasks across nine spatio-temporal subtypes. It reports instance distributions by difficulty and subtype and evaluates performance with a difficulty-weighted Pass@1 rate.

C.2 Detailed Information of STT-Arena

  • 227 manually verified instances span solvable and impossible tasks across nine spatio-temporal subtypes.Three annotators check query–checklist consistency, checklist correctness, and the plausibility of the spatio-temporal dynamics.Source: §C.2 · p. 14
  • Figures 9 and 10 present sample distributions across difficulty levels, solvability, and the nine spatio-temporal subtypes.Further descriptions and examples for difficulty levels and subtypes are provided in Tables 5 and 4, respectively.Source: §C.2 · p. 15, Figure 10 · p. 15, Figure 9 · p. 15
  • Pass@1 is the evaluation metric, and overall performance is the weighted average of results across the four difficulty levels.The weights are 71/227 for Easy, 67/227 for Medium, 59/227 for Hard, and 30/227 for Impossible.Source: §C.2 · p. 15
  • 71 Easy (31.3%), 67 Medium (29.5%), 59 Hard (26.0%), and 30 Impossible (13.2%) instances make up the benchmark.Subtype counts range from 38 for S1 to 17 for S3.Source: Figure 9 · p. 15

C.3 Detailed Information of Training Data

Training data for SFT and online RL is generated through a three-stage pipeline, producing validated SFT trajectories and RL instances.

C.3 Detailed Information of Training Data

  • The pipeline produces 2,212 validated SFT trajectories and 8,119 online RL instances.Not all generated RL instances are used in RL training.Source: §C.3 · p. 15
  • Qwen-3.5-397B-A17B generates the training data through a three-stage pipeline modeled on the STT-Arena construction pipeline.The passage identifies the same pipeline structure as benchmark construction.Source: §C.3 · p. 15
  • The 8,119 generated RL instances are not all used for RL training.Source: §C.3 · p. 15

C.4 Implementation Details of SFT and RL

SFT and online RL training use specified model and optimization settings, with data distributed across difficulty levels and conflict subtypes. The task taxonomy distinguishes nine conflict mechanisms and four solvability levels.

C.4 Implementation Details of SFT and RL

  • SFT uses Qwen-3-4B-Base for two epochs with a cosine scheduler, a 1.0 × 10−5 learning rate, 0.03 warmup ratio, 32K-token maximum sequence length, and effective global batch size 128.Training uses the LlamaFactory framework [Zheng et al., 2024].Source: §C.4 · pp. 15, 17
  • The taxonomy organizes nine conflict types into Temporal (T1–T3), Spatial (S1–S3), and Spatio-Temporal (ST1–ST3) categories.Each type has a distinct disruption mechanism and a real-world example.Source: Table 4 · p. 16
  • Task levels range from Easy, with one isolated conflict and corrective action, to Hard, with long-horizon cascading constraints requiring global replanning.Impossible tasks have no valid completion path; the correct response is to recognize and communicate infeasibility.Source: Table 5 · p. 16
  • Figures 11 and 12 show SFT trajectory and RL task distributions across difficulty levels and spatio-temporal subtypes.Source: Figure 11 · p. 16, Figure 12 · p. 17
  • Online RL fine-tunes the SFT checkpoint with REINFORCE++ in ROLL [Wang et al., 2025b], using a KL coefficient of 0.1 and learning rate 1.0 × 10−6.Each step samples 32 tasks and rolls out four trajectories per task, for up to 100 steps; trajectory length is capped at 32K tokens and generation at 4K tokens per step.Source: §C.4 · p. 17
  • All SFT and online RL training experiments run on four NVIDIA H200 GPUs.Source: §C.4 · p. 17

D.1 Cases of STT-Arena Construction Pipeline

The construction examples show a logistics environment with information-query and operational functions, a scenario blueprint, user profile, checklist, and injected dynamic triggers.

D.1 Cases of STT-Arena Construction Pipeline

  • The static environment example exposes information queries for vehicle status, depot maintenance, shipment journey status, route stops, goods inventory, and clinic schedules.The environment is represented as a Python class.Source: §D.1 · p. 17, Figure 13 · p. 18
  • The medium-difficulty T2 insulin-delivery task requires shipping insulin from HUB-NORTH to CLINIC-DOWNTOWN before 2024-08-15T08:00:00, while ensuring it never exceeds 8°C.The normal flow creates a shipment, assigns it to a refrigerated vehicle, defines route stops, checks clinic receiving hours, and marks the vehicle in transit.Source: §D.1 · p. 18
  • The sample user profile sets the goal of delivering insulin to CLINIC-DOWNTOWN before 10:00 and identifies the user as a temperature-sensitive-delivery logistics coordinator.Source: §D.1 · p. 19, Figure 15 · p. 20
  • A checklist evaluates task completion, and an example Python function validates an individual checklist item.Source: Figure 16 · p. 21, Figure 17 · p. 21
  • The dynamic environment example injects spatio-temporal triggers into the static environment.The example class maintains environment state and internal conflict-trigger tracking.Source: §D.1 · p. 21, Figure 18 · p. 22

D.2 Cases of Failure Mode

The cases illustrate three ways agents fail to recover from disruptions: executing from stale state, misdiagnosing a trigger, and stopping without verifying task completion.

Stale-State Execution

  • After an irrigation-schedule lookup fails, the agent keeps its prior assumption and makes 20+ retries with guessed identifiers instead of refreshing state and replanning.The agent had found that only V3 was assigned to Segment C and that V3 was already open.Source: §D.2 · p. 22, Figure 19 · p. 23

Misdiagnosis of Dynamic Triggers

  • A Schengen transit restriction is misread as formatting or availability trouble, leading to repeated booking attempts and a reroute through another Schengen hub.The alternative route remains subject to the same regulatory block.Source: §D.2 · p. 23, Figure 20 · p. 24

Missing Post-Adaptation Verification

  • After a warehouse outage, the agent reroutes to a fallback depot but stops after dispatch succeeds, leaving 180 doses of the 500-dose requirement unverified and unmet.The delivered quantity is 320, leaving the shortfall unresolved.Source: Figure 21 · p. 24

E Prompt Templates

The prompt templates specify procedures for curating and constructing executable environments and spatio-temporal tasks, validating their functionality and constraints, and evaluating agents, user simulators, and judges.

E.1.1 Two-Stage Filtering Prompt Template

  • The first filter retains actionable requests that depend on persistent, domain-specific state and require reading or changing that state.It excludes open-domain factual questions, hypotheticals, and isolated reasoning without interaction with persisted state.Source: §E.1.1 · p. 25
  • The second filter selects stateful tasks that can naturally encounter at least one concrete spatio-temporal conflict during a multi-step workflow.Its prompt considers temporal, spatial, and combined conflict mechanisms, while filtering out superficial temporal or spatial references.Source: §E.1.1 · p. 26, Figure 23 · p. 27

E.1.2 Prompt Template for Environment Synthesis

  • Environment synthesis infers a plausible, appropriately scoped domain from each filtered query and returns a structured summary, introduction, and feasibility metrics.The prompt asks for usefulness and modelability ratings to guide subsequent synthesis.Source: §E.1.2 · p. 28, Figure 24 · p. 29
  • Synthesis prompts then define entities, attributes, constraints, and categorized query and state-change operations for the environment.The prompts translate these specifications into typed Python classes, first with state attributes and then with tool methods that return structured success or failure dictionaries.Source: §E.1.2 · p. 28, Figure 25 · p. 29, Figure 26 · p. 30

E.1.3 Functional Validation Prompt Template

  • Functional validation prompts generate initialization configurations whose structure, types, constraints, and cross-entity references match the candidate environment class.The generated data should include diverse valid states and realistic values, avoiding placeholder-like names and using ISO dates or timestamps.Source: §E.1.3 · p. 33
  • A separate validation prompt generates sequences of tool calls to test candidate static environments and filter out environments that fail functional checks.The procedure first generates test configurations, then generates validation calls against them.Source: §E.1.3 · p. 33
  • Functional-validation prompts generate diverse configurations and exploratory tool calls covering positive, negative, and boundary cases across all interfaces.The prompt describes one tool invocation per testing round and uses the environment return plus a backend pass, warning, or fail evaluation.Source: Figure 29 · p. 34, §E.1.3 · p. 34, Figure 30 · p. 35

E.2.1 Conflict Assignment and Blueprint Generation Prompt Template

  • The conflict-assignment and blueprint prompts select environment-supported conflict types, then specify concrete field-level mutations, triggers, recovery paths, and difficulty.Feasibility depends on the environment supporting a valid activation trigger, conflict mechanism, and observable state update; blueprint rules require normal-flow operations from the environment and a read/query activation operation.Source: §E.2.1 · pp. 35–36, Figure 31 · p. 36, Figure 32 · p. 37

E.2.2 Dynamic Environment Construction Prompt Template

  • Dynamic-environment prompts inject deterministic conflict triggers and ground blueprints into executable tasks with consistent entity IDs, mutations, user requests, and user profiles.The user profile specifies communication style, flexibility, and clarification responses tailored to whether a task is solvable or impossible.Source: §E.2.2 · p. 37, Figure 33 · p. 38, Figure 34 · p. 39

E.3.1 Checklist Generation Prompt Templates

  • The checklist-generation prompt creates conflict-aware success criteria and Python checks for conflict detection, recovery-path execution, and final-state validity.This template is part of the dual-agent assessment stage.Source: §E.3 · p. 40, Figure 36 · p. 41

E.3.2 Dual-Agent Validation Prompt Templates

  • Dual-agent validation has a plan agent generate a blueprint-guided tool sequence reaching the conflict, while a check agent verifies trigger timing, state mutation, and original-plan failure.The plan includes follow-up calls to observe post-trigger state changes.Source: §E.3.2 · p. 41, Figure 37 · p. 42, Figure 38 · p. 43

E.3.3 Consistency Check Prompt Templates

  • A consistency auditor checks mutual coherence across task artifacts, including the query, initial configuration, conflict design, mutations, checklist, and difficulty label.It returns a structured JSON verdict with issue descriptions and focuses on contradictions, missing links, and implausible transitions.Source: §E.3.3 · p. 43, Figure 39 · p. 44

E.4 Prompt Templates of STT-Arena Evaluation

  • Evaluation prompts require the tested model to make exactly one tool call per turn, while separate prompts govern the passive user simulator and an LLM judge for impossible tasks.The user simulator follows the user profile, reveals withheld details only upon direct query, and signals completion when satisfied; the judge returns structured binary verdicts with supporting evidence.Source: §E.4 · p. 44, Figure 40 · p. 45, Figure 41 · p. 45
Loading 2605.18548v1…