PaperlayerView exact revision on arXiv

Source-linked AI summary

Can Agents Generalize to the Open World? Unveiling the Fragility of Static Training in Tool Use

Song-Lin Lv, Weiming Wu, Rui Zhu, Zi-Jian Cheng, Lan-Zhe Guo

arXiv:2607.01084v1cs.AI

TL;DR

LLM agents that perform well on static benchmarks may fail when queries, tools, observations, and domains shift in deployment. The paper formalizes OpenAgent, evaluates these shifts in a controlled four-tier sandbox, and finds degradation in both SFT and RL agents while introducing PAFT to improve SFT robustness.

  • Problem

    Static benchmark success does not establish generalization when real-world queries, tool sets, and interaction dynamics are non-stationary.

    Source: §1 · p. 1

  • Method

    The paper formalizes OpenAgent across query, action, observation, and domain shifts and evaluates agents in a controlled sandbox spanning Perception, Interaction, Reasoning, and Internalization.

    Source: §1 · p. 1

  • Results

    Both SFT and RL agents degrade under open-world shifts, with SFT showing trajectory overfitting and symbolic anchoring and RL showing boundary blindness despite better semantic grounding.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    PAFT uses controlled observation anomalies and symbolic noise to mitigate SFT trajectory overfitting and support robust generalization in open environments.

    Source: §1 · p. 2, §7 · p. 9

    The study emphasizes controlled evaluation and refusal behavior rather than unrestricted deployment, while noting risks from stronger tool-use agents in automated decision-making systems.

    Source: Impact Statement · p. 10

Abstract

from arXiv · show

While Large Language Model (LLM) agents demonstrate proficiency in static benchmarks, their deployment in real-world scenarios is hindered by the dynamic nature of user queries, tool sets, and interaction dynamics. To address this generalization gap, we formalize OpenAgent (Tool-Use Agent in Open-World), a problem setting characterized by distributional shifts across query, action, observation, and domain dimensions. To systematically diagnose its impact, we construct a controlled sandbox environment where we define fine-grained environmental shifts across a four-tier hierarchy, Perception, Interaction, Reasoning, and Internalization, and conduct a comprehensive series of experiments. Our analysis yields a series of key insights, demonstrating that agents trained via both Supervised Fine-Tuning(SFT) and Reinforcement Learning suffer from varying degrees of performance degradation when confronting open environmental shifts. Building on these insights, we propose Perturbation-Augmented Fine-Tuning, a disturbance-based intervention strategy for SFT that lays the foundation for enhancing agent robustness and utility in realistic environments. Our code will be released at: https://github. com/LAMDA-NeSy/OpenAgent.

1. Introduction

Static benchmarks can conceal serious fragility when queries, tools, observations, and task domains shift. This paper formalizes OpenAgent, evaluates these shifts in a controlled sandbox, identifies distinct SFT and RL failure modes, and proposes PAFT for SFT robustness.

  • OpenAgent formalizes open-world tool use through shifts in user queries, tool sets, interaction dynamics, and task domains.Source: §1 · p. 1
  • A controlled sandbox preserves a closed-set baseline while injecting perturbations across Perception, Interaction, Reasoning, and Internalization.Source: §1 · pp. 1–2
  • Both SFT and RL agents degrade under open-world shifts, but SFT exhibits trajectory overfitting and symbolic anchoring whereas RL shows boundary blindness despite better semantic grounding.Source: §1 · pp. 1–2
  • PAFT injects controlled observation anomalies and symbolic noise into SFT trajectories to reduce shallow pattern memorization and restore open-world generalization.Source: §1 · p. 2
  • The paper’s contributions are formalizing OpenAgent, systematically evaluating robustness, diagnosing failure modes, and proposing PAFT.Source: §1 · p. 2

2. Related Works

Related work spans SFT, RL, and hybrid post-training paradigms for tool-use agents, alongside local, remote, and increasingly realistic evaluation benchmarks.

  • Post-training methods include SFT approaches that standardize tool invocation, RL frameworks that use rewards for decision-making, and hybrids combining SFT initialization with RL exploration.Source: §2 · p. 2
  • Agent evaluations cover local tools and remote protocols, while existing benchmarks emphasize task success and robustness studies increasingly examine inference-time perturbations and realistic environments.Source: §2 · p. 2

3. Preliminaries

The paper models tool use as sequential decision-making under an agent–environment loop, where distribution shifts can cascade through later actions. OpenAgent organizes these shifts across query, action, observation, and domain dimensions.

  • An agent selects actions from history containing the query, prior actions, and observations, and the environment returns observations until task completion.Source: §3.1 · p. 3
  • OpenAgent differs from passive input perturbations because a shifted observation can alter all subsequent decisions and compound trajectory errors.Source: §3.2 · p. 3
  • Query shifts change semantic directives or linguistic forms, while action shifts include renamed tools, altered semantics, and reconfigured dependencies.Source: §3.2 · p. 3
  • Observation shifts introduce changed formats, errors, null values, or corrective instructions, requiring agents to adjust policy or recognize terminal failure.Source: §3.2 · p. 3
  • Domain shifts jointly change queries, tools, and observations while preserving latent reasoning structure that must be re-grounded in the target domain.Source: §3.2 · pp. 3–4

4. Evaluation Tasks

The evaluation uses a controlled synthetic sandbox, four complementary metrics, and task perturbations spanning perception, interaction, reasoning, and internalization capabilities.

Experimental Setup

  • The synthetic sandbox isolates environmental variables against a closed-environment baseline using geographical POI and calculation tasks.Source: §4.1 · p. 4

Metrics

  • The study measures correctness, efficiency, robustness, and adaptability through TER, AES, ATCL, and RR.Source: §4.1 · p. 4

Perception

  • Perception tests instruction robustness, schema adaptability, and semantic grounding through blurred or redundant instructions, changed parameter keys, and symbolic or semantic tool-name perturbations.Source: §4.2 · p. 4

Interaction

  • Interaction tests format parsing, error recovery, and feedback-conditioned adjustment using altered return formats, explicit or null failures, and value or tool redirection.Source: §4.3 · p. 5

Reasoning

  • Reasoning evaluates whether models derive changed tool semantics rather than memorize rules, including a calculation shift that changes a tool’s functional factor while preserving its interface.Source: §4.4 · p. 5

5. Key Observations

Across four diagnostic tiers, SFT and RL both degrade under open-world shifts, but their strengths and failures diverge: RL adapts better locally, while both struggle with inverted dependencies and unsolvable tasks.

Tier 1: Perception

  • SFT relies on brittle symbolic anchoring, whereas RL maintains stronger semantic grounding and lower error under shifted tool semantics.As surface names are perturbed, SFT increasingly invokes incorrect tools; RL also decays with overfitting but retains higher accuracy.Source: §5 · p. 6, Figure 4 · p. 7

Tier 2: Interaction with Guidance

  • Under ambiguous feedback, both paradigms degrade, but explicit guidance exposes RL’s stronger policy adaptation and SFT’s trajectory inertia.SFT often interprets corrective observations as successful outcomes, while RL is encouraged to treat environmental feedback as a state update.Source: §5 · pp. 6–7, Figure 5 · p. 7

Tier 3: Reasoning Generalization

  • RL generalizes better to local adaptation, but both SFT and RL suffer substantial degradation when tool dependencies are globally inverted.In Logic Inversion, both models often hallucinate the training-time topology despite explicit documentation of the reversed prerequisite order.Source: §5 · p. 7

Tier 4: Boundary Awareness

  • Both methods show limited refusal for unsolvable queries, with SFT ignoring fatal feedback and RL recognizing failure but still forcing completion.The distinct mechanisms reflect SFT’s perceptual failure and RL’s outcome-oriented decisional bias toward answering at all costs.Source: §5 · pp. 7–8, Figure 7 · p. 8

Cross-tier comparison

  • At stable training, SFT degrades across perturbations, while RL’s drops are concentrated mainly in logic inversion and query variations.Both methods remain robust to simple format changes, but SFT shows significant degradation under domain transfer.Source: Figure 6 · p. 8, Figure 7 · p. 8

6. Perturbation-Augmented Fine-Tuning

PAFT augments clean SFT trajectories with controlled environmental, solvability, and symbolic perturbations to improve open-world robustness. Across tiers and training stages, it consistently reduces degradation, with targeted components addressing distinct failure modes.

6. Perturbation-Augmented Fine-Tuning

  • PAFT augments clean trajectories with controlled perturbations that expose anomalous observations, unsolvable states, and symbolic variants while supervising corrective continuations.Unlike input-only augmentation, the method injects perturbations mid-trajectory and expands the support of training histories.Source: §6 · p. 8

6.1. Method

  • Environmental Feedback Perturbation trains models to recover from unexpected feedback, converting open-loop execution into closed-loop corrective behavior.It injects anomalies such as tool redirection or value deprecation and supervises a corrective action rather than assuming success.Source: §6.1 · p. 8
  • Solvability Boundary Perturbation teaches explicit refusal by adding fatal-error trajectories, breaking the assumption that every problem is solvable.The resulting refusal action halts execution when no alternative tool can recover the task.Source: §6.1 · pp. 8–9
  • Symbolic Representation Perturbation improves perception and reasoning by rewriting tool names and documentation so decisions rely on functional semantics rather than memorized tokens.The component targets symbolic anchoring and associated downstream reasoning errors through surface-level representation changes.Source: §6.1 · p. 9, §6.2 · p. 9

6.2. Experimental Results

  • PAFT yields consistent improvements across all training stages and open-environment tiers, transforming early-stage degradation into positive deltas across Tiers 1–3.The evaluation uses Accuracy Delta for Tiers 1–3 and absolute RR for Tier 4, with α = 0.3 as the default perturbation ratio.Source: §6.2 · p. 9, Table 1 · p. 9
  • Component ablations show targeted complementarity: EFP mainly improves Tier-2 adaptability, SBP restores Tier-4 refusal, and SRP primarily strengthens Tiers 1 and 3.Each component is individually insufficient, supporting PAFT as a diagnosis-driven intervention rather than generic augmentation.Source: §6.2 · p. 9

7. Conclusion

The paper formalizes OpenAgent and uses a controlled sandbox to diagnose tool-use adaptability under non-stationary environmental shifts. It finds vulnerabilities in both SFT and RL agents and proposes PAFT to mitigate SFT trajectory overfitting.

7. Conclusion

  • The study formalizes OpenAgent and uses a controlled sandbox to diagnose LLM-agent adaptability under open-world shifts.Source: §7 · p. 9
  • Both SFT and RL paradigms exhibit vulnerabilities when facing non-stationary environmental shifts.Source: §7 · p. 9
  • PAFT is proposed as a disturbance-based intervention strategy designed to mitigate trajectory overfitting in SFT.Source: §7 · p. 9

Impact Statement

The work studies robustness and failure modes of tool-use agents under open-world shifts. Its controlled evaluation emphasizes refusal and boundary awareness while using a sandboxed, tool-mediated environment to examine reliability and safety.

Impact Statement

  • The work studies robustness and failure modes of tool-use agents under open-world shifts, including conditions involving ignored feedback, tool misuse, and hallucinated answers.Source: Impact Statement · p. 10
  • The sandboxed evaluation emphasizes refusal behavior, boundary awareness, and controlled testing rather than unrestricted deployment.Source: Impact Statement · p. 10, A. Sandbox Environment · p. 15
  • The tool environment uses anonymized Beijing POI data and constrained tool interfaces to reduce data leakage and structure tool-use evaluation.Source: §A.1 · p. 15, §A.2 · p. 16
  • The setup supports multi-turn interactions with one tool call per conversation turn during training and evaluation.Source: §A.3 · pp. 16–17
  • The experimental environment combines a query database, sandbox code execution, training and test sets, and tool-call interaction logic.Source: A. Sandbox Environment · p. 15, §A.3 · p. 17

A.4. Training and Test Data

The dataset is generated from symbolic multi-step tool-use trajectories and converted into natural-language queries, with disjoint databases separating training and test entities. Its one-to-six-step design and varied tools and scenarios target semantic grounding and long-range tool planning.

  • Symbolic trajectories sample initial conditions and target answers, chain tool invocations, and produce final answers before conversion into natural-language question-answer pairs.Source: §A.4 · pp. 17–18
  • 6,930 samples are split into N = 6,050 training and N = 880 test examples for evaluation.Source: §A.4 · p. 18
  • Training and test sets share tool-chain length distributions but use disjoint POI database subsets, requiring generalization to unseen entities rather than attribute recall.Source: §A.4 · p. 18
  • Tasks span one to six steps, with two-step tasks comprising 28.4% and nearly 40% involving three or more reasoning steps.Source: §A.4 · p. 18
  • The dataset combines caller-ID, ticket-price, and geographic-measurement tools across diverse scenarios to encourage semantic grounding and robust path planning.Source: §A.4 · p. 18

B. Experimental Details

Experiments compare fully trained SFT and RL agents under controlled inference conditions while introducing query, schema, naming, and observation-format perturbations. The setup preserves a common tool-use protocol and tests whether agents rely on functional semantics rather than fixed symbols or formats.

  • SFT uses full-parameter fine-tuning, while RL uses GRPO tool calling; both models are fully fitted with controlled data amounts for fair comparison.Source: B. Experimental Details · pp. 18–19
  • Inference uses temperature 0.0 and Top-p 1.0, with three tool-setting experiments analyzed under identical sampling conditions.Source: B. Experimental Details · p. 19
  • The protocol requires one tool call per turn, strict JSON tool formatting, and final answers wrapped in <answer></answer>.Source: B. Experimental Details · p. 19
  • Instruction Blurring and Information Redundancy vary linguistic form, ambiguity, and irrelevant context to test intent reconstruction and information filtering.Source: §C.1.1 · p. 20
  • Schema and semantic perturbations change argument keys, tool names, descriptions, identifiers, or return formats to test adaptation beyond string matching and rigid JSON parsing.Source: §C.1.2 · p. 20, §C.1.3 · p. 20, §C.2.1 · p. 21

C.2.2. ERROR CORRECTION. •

The study evaluates whether agents can use altered observations, tool failures, changed rules, shorter tool paths, reversed dependencies, insufficient conditions, and domain transfers to revise their actions. These tests span interaction, reasoning, refusal, and domain-shift robustness.

ERROR CORRECTION

  • Error Return, Null Return, redirection, and value changes test whether agents detect explicit or silent failures and adapt actions using feedback.Source: §C.2.2 · p. 21, §C.2.3 · p. 21

Rule Reasoning

  • Calculation Shift changes tool semantics while preserving interfaces, testing whether agents derive results from current descriptions instead of memorized rules.Source: §C.3.1 · p. 21

Reasoning

Active Refusal

  • Active Refusal creates Missing Condition and Fatal Error cases in which agents should recognize information deficiency or operational deadlock instead of hallucinating answers.Source: §C.4.1 · p. 22, Table 13 · p. 22

Domain Shift

  • Domain Shift transfers evaluation from geographic lookup to medical registration while preserving latent causal structure and changing queries, tools, and observations.Source: §C.4.2 · p. 22, Table 14 · p. 23

D.2. Tier 2 Experiments

Across open-world tests, RL generally adapts better than SFT to altered tools, observations, rules, and domains, although both models struggle with some unsolvable or causally inverted tasks. The experiments expose SFT’s semantic anchoring and feedback-insensitivity alongside RL’s stronger adjustment behavior.

Tier 2 Experiments

Tier 3 Experiments

  • RL outperforms SFT in trace simplification and rule reasoning by adopting composite tools and updated calculation logic instead of reverting to memorized sequences or constants.Source: §D.3 · p. 34, §E.3.1 · p. 41, §E.3.2 · p. 41

Tier 4 Experiments

  • RL-280 retains 96.15% of closed-environment accuracy after domain transfer, compared with 71.96% for SFT-800, while SFT produces more invalid tool calls.Source: §D.4 · p. 37, Table 32 · p. 37

Tier 1 Experiments

  • SFT shows semantic anchoring under renamed or adversarial tools, whereas RL can sometimes invoke new tools and adjust to changed return-value instructions.Source: §E.1.3 · p. 38, §E.2.2 · p. 39

E.4.1. FATAL ERROR

Fatal errors expose a sharp difference between SFT and RL: SFT agents answer despite unusable or missing information, whereas RL agents may recognize the problem but inconsistently refuse or speculate. Real-API validation preserves these perturbation trends across interaction, perception, format, and refusal evaluations.

Fatal Error

  • After the distance tool returns an unavailable-tool error, SFT directly reports a distance, while RL either hypothesizes 10000.0 meters or states that calculation cannot be performed.The SFT response reports 10343.2 after the tool failure; RL produces either a hypothetical estimate or a refusal.Source: §E.4.1 · pp. 41–42
  • SFT directly gives hypothetical answers to every question, while RL sometimes refuses and sometimes provides hypothetical outputs when information is missing.Examples include SFT answering after a fatal tool error and RL either assuming a distance or correctly refusing.Source: §E.4.1 · pp. 41–42, §E.4.2 · p. 42
  • For an unsuccessful administrative query, RL attempts a tool call and then returns Unknown rather than fabricating a district name.The query produces a “Record not found” observation before the refusal-like answer.Source: §E.4.2 · p. 42
  • RL can still execute valid tool-use chains under changed inputs, including calculating a distance from returned observations and transferring the workflow to a medical-location domain.The examples show a returned distance of 8311.98 and successful retrieval of medical-facility coordinates.Source: §E.4.2 · p. 42, §E.4.3 · pp. 42–43
  • Replacing the sandbox distance tool with Amap’s real driving-distance API tests both SFT and RL on live routing traces, formats, error codes, and latency characteristics.The validation covers Tier-2 interaction perturbations, Tier-1 perception perturbations, Tier-2 format adaptability, and Tier-4 fatal-error refusal rate.Source: F. Real-World API Validation · p. 43, Figure 10 · p. 43

Related papers

Loading 2607.01084v1…