Don't Just Fine-tune the Agent, Tune the Environment

Motivation

• High-quality multi-turn tool-use data are scarce, while BFCL V3 spans 8 domains and 84 tools and requires success across every interaction turn.These constraints make conventional data-driven training difficult.Source: §1 · p. 2
• Static synthetic SFT trajectories often fail to generalize, whereas standard RL faces cold-start exploration and instability on long interaction chains.The agent may remain trapped in low-quality rollouts and later suffer performance collapse.Source: §1 · p. 2

Approach

• ENVIRONMENT TUNING combines a structured curriculum, actionable environment augmentation, and fine-grained progress rewards to address scarcity, exploration, and stability.The framework turns failures into corrective learning signals and replaces sparse outcomes with continuous task-completion feedback.Source: §1 · p. 2

Contribution

• Without expert demonstrations, ENVIRONMENT TUNING shifts learning from trajectory-based imitation to environment-based exploration.Agents learn multi-turn tool-use capabilities directly from problem instances.Source: §1 · p. 2

Results

• With only 400 training samples, the method lifts base Qwen2.5-7B from near-zero to strong in-distribution performance, boosts SFT-tuned watt-tool-8B to 54.25%, and nearly doubles ToolACE-2’s ACEBench score from 8.5% to 15.0%.The passage describes these results as evidence of robust generalization where SFT often fails.Source: §1 · p. 3

Tool-integrated reasoning

• Tool-integrated reasoning commonly uses RL to teach strategic tool invocation, but trajectory-level optimization can become unstable and collapse.Existing remedies include fine-grained credit assignment, entropy-guided exploration, and trajectory filtering.Source: §2 · p. 3
• Many prior methods are evaluated with one tool or a small homogeneous toolset, unlike tasks requiring orchestration of large, diverse tool ecosystems.This paper targets complex, stateful tasks spanning multiple interaction turns.Source: §2 · p. 3

Multi-turn tool orchestration

• BFCL and ACEBench evaluate multi-turn orchestration of diverse APIs, but their realistic, human-annotated data are high-quality and scarce.The benchmarks therefore expose a substantial training-data challenge.Source: §2 · p. 3

Training strategies

• Synthetic SFT generates large tool-use trajectory corpora, while online RL learns through environment interaction but has produced only modest BFCL gains.The limited gains motivate more efficient learning paradigms for complex environments.Source: §2 · p. 3

3.1 Preliminary and challenges

• Multi-turn tool use requires both precise tool-call syntax and high-level planning, with failures including invalid parameters, nonexistent tools, and environment corruption.Direct single-stage RL on Qwen2.5-7B-Instruct collapsed within 70 steps in one 400-instance experiment.Source: §3.1 · pp. 3–4

3.2 Structured curriculum RL training

• The four-stage curriculum progresses from foundational syntax and tool-call correctness to base tasks, complex scenarios, and evaluation-aligned training.Stage transitions occur after validation accuracy plateaus and gradient norms stabilize.Source: §3.2 · pp. 4–5
• Stage 1 uses format and tool-call rewards to rapidly master valid outputs before task-oriented reasoning.The specialized reward is reported to make syntax mastery a prerequisite for efficient learning in later stages.Source: §3.2 · p. 4

3.3 Actionable environment augmentation

• Actionable Environment Augmentation replaces generic errors with pedagogical hints that reveal inter-tool dependencies and environment-specific constraints.This supports guided discovery rather than memorization of preconstructed dependency graphs.Source: §3.3 · p. 6
• Augmented feedback turns failed turns into explicit corrective guidance, enabling agents to diagnose root causes and discover strategies unavailable under ambiguous baseline feedback.The approach is illustrated in file-system and multi-API travel cases.Source: §3.3 · p. 6

3.4 Fine-grained progress reward

• The Progress Reward evaluates each turn using environment-state correctness and tool-execution correctness, then averages successful turns across the episode.This distinguishes partially successful trajectories from completely wrong ones and provides denser feedback than terminal binary reward.Source: §3.4 · p. 6
• Table 1 reports that ENVIRONMENT TUNING boosts all base models on BFCL V3 and achieves competitive results against proprietary models using 400 training samples.The table summarizes the main in-distribution benchmark comparison.Source: Table 1 · p. 7

4.2 Main results

• Environment Tuning improves in-distribution performance across base models and SFT-tuned models, including Qwen2.5-7B-Instruct’s increase from 7.00% to 36.92%.It also raises ToolACE-2-Llama-3.1-8B to 47.18% and watt-tool-8B to 54.34%.Source: §4.2 · p. 8
• OOD evaluation shows SFT baselines can collapse, with xLAM-2 falling from 70.50% on BFCL V3 to 5.00% on Web Search, whereas Environment Tuning improves generalization.For example, Llama-3.1-8B-Instruct rises from 1.00% to 15.00% on Web Search.Source: §4.2 · pp. 8–9
• Environment Tuning outperforms direct RL on complex OOD tasks, raising ToolACE-2-Llama-3.1-8B from 8.34% to 15.00% on ACEBench while direct RL drops it to 6.65%.The comparison is reported for ACEBench Agent within the OOD evaluation.Source: §4.2 · p. 9

4.3 Ablation study

• The structured curriculum reaches 36.92%, a 19.50% increase over direct GRPO, while direct RL from the start yields minimal gains.The ablation uses Qwen2.5-7B-Instruct and a 400-sample dataset.Source: §4.3 · p. 9, Table 3 · p. 9
• Environment augmentation produces more stable learning and over 20% improvements on Missing Parameters and Missing Functions splits.The effect is especially pronounced on these complex, ambiguous scenarios.Source: Figure 4 · p. 9, §4.3 · p. 10
• Progress Reward matters increasingly with task complexity: binary rewards suffice for foundational Base data but fail near zero on complex splits.Progress Reward also yields more stable and effective learning on Long Context.Source: §4.2 · p. 9, §4.3 · p. 10

5 Conclusion

• Environment Tuning combines environment-driven exploration, structured curriculum, actionable augmentation, and progress rewards to improve learning under extreme data scarcity.The conclusion reports substantial in-distribution gains and stronger out-of-distribution generalization than supervised baselines.Source: §5 · p. 10

A Sequential Multi-turn Decision-Making Model.

• A multi-turn episode contains sequential user requests, with tool calls and final answers completing each turn before the next request is revealed.The episode ends only after the final answer to the last request.Source: A Sequential Multi-turn… · p. 14
• The agent receives a sparse terminal reward only after the complete episode, making credit assignment and exploration difficult in long-horizon tasks.The terminal reward is binary: RT ∈ {0, 1}.Source: A Sequential Multi-turn… · p. 14

B Implementation of Actionable Environment Augmentation

• An Augmentation Generator analyzes error trajectories and produces enhanced feedback, while a Constitutional Judge checks its quality and exploration constraints.The workflow uses few-shot examples and can retry rejected feedback up to three times.Source: B Implementation of Actionable… · p. 14, §B.3 · p. 18

B.1 Augmentation Generator Prompt

• The generator classifies error needs and creates guidance that clarifies root causes without prescribing exact solutions or steps.Its categories include missing dependencies, parameter constraints, state prerequisites, format mismatches, and unavailable resources.Source: §B.1 · p. 15

B.2 Constitutional Judge Prompt

• The Constitutional Judge approves feedback only when it avoids solution leakage and over-constraining while remaining actionable and high quality.It checks whether guidance preserves exploration space and helps diagnose tool-usage problems.Source: §B.2 · pp. 16–17

B.3 Workflow and Practical Considerations

• Only 3–5 domain-specific few-shot examples are needed to bootstrap automated feedback generation for hundreds of error scenarios.This reduces manual engineering for new domains.Source: §B.3 · p. 18

C Detailed Progress Reward Components

• Progress Reward scores each turn as successful only when both the environment state or execution result is correct, then averages successful turns across the episode.State comparisons evaluate persistent changes, while return-value comparisons evaluate information retrieval and other transient outputs.Source: C Detailed Progress Reward Components · pp. 18–19

D.1 Training Dataset

• BFCL V3 contains 800 multi-turn samples across four 200-sample splits, with 400 selected for training and 400 reserved for held-in testing.The splits are Base Multi-Turn, Missing Parameters, Missing Functions, and Long-Context.Source: §D.1 · p. 19
• BFCL V4 Web Search and Memory tracks test OOD generalization using tasks such as multi-hop retrieval, synthesis, and long-context memory across multiple backends.Web Search also includes probabilistic network failures.Source: §D.1 · p. 19
• τ2-bench evaluates collaborative customer-service agents across retail, airline, and telecom domains under shared-tool policies.Both the agent and a user simulator can operate on the shared tools.Source: §D.1 · p. 19
• ACEBench Agent tests advanced generalization through dynamic multi-turn dialogues requiring multi-step reasoning, long-term context management, and implicit-rule adherence.Evaluation occurs in a sandboxed environment designed to resemble real-world interaction.Source: §D.1 · p. 19

D.2 Training Details

• The adapted PPO objective combines decoupled clipping with a KL divergence penalty to improve stability and exploration.Source: §D.2 · pp. 19–20
• The advantage estimate is computed without a critic by normalizing terminal rewards within groups of trajectories generated for the same prompt.The group statistics are the reward mean and standard deviation.Source: §D.2 · p. 20
• The implementation uses β = 0.1 for KL regularization and clipping values ϵlow = 0.2 and ϵhigh = 0.28.Source: §D.2 · p. 20

E.1 Training Dynamics

• Stage 1 rapidly increases format and tool-call rewards, saturates Is Tool Call at 1.0, and reduces interaction rounds.These changes indicate mastery of output syntax and tool schemas while eliminating void turns and wasted attempts.Source: §E.1 · p. 20, Figure 5 · p. 20
• The full curriculum keeps gradient norms stable throughout training, preventing the catastrophic explosions associated with long-horizon reinforcement learning.Source: §E.1 · pp. 20–21
• Steady validation improvement and stable gradient norms across four stages show that the curriculum supports learning without training collapse.Figure 5 uses a held-out validation set of 100 samples from the remaining BFCL data.Source: Figure 5 · p. 20, §E.1 · p. 21

E.2 Training Instability in Single-Stage RL

• Around step 70, single-stage RL accuracy collapses as the gradient norm explodes after an initial 10–15% gain over the base model.The setup uses 400 BFCL V3 training instances without the proposed curriculum or environment augmentation.Source: §E.2 · p. 21, Figure 6 · p. 21

E.3 KL Loss Coefficient

• β = 0.1 maintains policy entropy and sustained accuracy, whereas β = 0.001 only delays collapse and no KL penalty permits entropy collapse.Source: §E.3 · p. 21, Figure 7 · p. 22

F Prompt Templates

• The prompt templates require tool calls in a structured JSON array and reserve answer responses for completed, impossible, or underspecified requests.Source: F Prompt Templates · p. 23

G Case Study

• The case studies compare a minimally informative baseline environment with a tuned environment that supplies rich, actionable feedback for diagnosing and recovering from errors.Source: G Case Study · p. 24

G.1 File System Management Scenario

• Actionable feedback identifies that absolute paths are disallowed, enabling a directory change before removing the file and parent directory.Generic “No such file or directory” feedback instead causes an incorrect diagnosis and premature abandonment.Source: §G.1 · pp. 24–25
• After deleting the located file, the directory-removal attempt fails when the file path is omitted and absolute paths are rejected by the rm function.Source: §G.1 · pp. 24–25
• The successful trajectory changes into SuperResearch, removes findings_report, returns to the parent directory, and removes SuperResearch.Source: §G.1 · p. 25
• Listing with the a parameter returns an empty current directory, indicating that visible and hidden files are absent.Source: §G.1 · pp. 25–26
• Attempts to grep draft files fail because the searched paths do not exist, prompting a request to verify filenames or locations.Source: §G.1 · pp. 26–27
• The copy-and-rename operation for summary_draft.docx also fails when the source file is absent from the current directory.Source: §G.1 · pp. 27–28

Environment feedback

• Actionable feedback identifies malformed airport parameters and suggests retrieving the correct code, whereas vague baseline feedback leads the agent to abandon the flight task.The tuned environment distinguishes an invalid code from an unavailable route and supports successful self-correction.Source: §G.2 · pp. 28, 30, 32

Data limitation

• The scenario shows that airport codes alone cannot determine which listed airport is closest to Crescent Hollow without names or geographical details.The available list provides codes, but the agent states that further location information or another capability is required.Source: §G.2 · p. 29

Multi-step travel recovery

• The tuned environment converts an invalid Pinehaven airport parameter into a corrective workflow that finds PHW and returns a $3800.00 business-class fare from CWH.The agent first calls list_all_airports, then get_nearest_airport_by_city, and finally get_flight_cost for March 3, 2024.Source: §G.2 · pp. 30–31

Social-media API

• The agent successfully posts a flight-related tweet after authentication, but an initial call fails because post_tweet does not accept a username argument.Removing the extraneous username allows the tweet to be posted successfully for michael_t.Source: §G.2 · pp. 33–34
• The retweet workflow retrieves the recent tweet ID before reporting that the tweet was already retweeted.The scenario therefore produces no additional retweet action.Source: §G.2 · p. 34

Travel and social-media chaining

• Actionable feedback identifies that estimate_distance requires zipcodes, enabling retrieval of 94016 and 74532 and a successful 880.0-kilometer estimate.The tuned environment recommends verifying numeric zipcodes, after which the agent calls get_zipcode_based_on_city and estimate_distance.Source: §G.3 · pp. 35–37

Environment feedback

• Generic baseline errors leave the agent stuck in failed calls and parameter mismatches, whereas precise hints support repairing incomplete plans and completing multi-step objectives.The comparison contrasts “distance not found in database” with guidance that names the input-type mismatch and corrective action.Source: §G.3 · pp. 35–36

Travel and social-media chaining

• The tweet was successfully posted with its content and tags, received ID 12, and was subsequently reported as successfully retweeted.Source: §G.3 · pp. 37–38

Vehicle control

• The vehicle environment prevents refueling beyond tank capacity and reports that the tank may already be full when a 50-gallon fill fails.The agent cannot infer a successful refill from the failed call and is directed to check the current fuel level manually.Source: §G.3 · p. 40
• All vehicle doors are locked, but engine startup fails because the brake pedal must be pressed.The response reports lockStatus as locked with zero remaining unlocked doors while retaining the startup prerequisite.Source: §G.3 · p. 40
• The mean of the four travel distances is 452.5 km.The mean function receives 750.0, 320.0, 450.0, and 280.0 km.Source: §G.3 · p. 41

• The case study is intended to show robust multi-step diagnostics rather than superficial pattern matching in an out-of-distribution telecom setting.Source: §G.4 · p. 41
• ENVIRONMENT TUNING resolves both underlying issues and reaches the user’s required excellent speed, unlike the pretrained agent, which resolves only Data Saver.The two issues are Data Saver mode and disabled device-level data roaming; the final test reports excellent speed after both are addressed.Source: §G.4 · pp. 42, 44
• The pretrained agent stops after speed improves to good, repeatedly retests, and fails to recognize disabled roaming as the remaining issue.It then incorrectly suggests changing the user’s plan despite the user’s explicit constraint.Source: §G.4 · pp. 42–43
• The tuned agent re-evaluates its initial diagnostics after the speed reaches only good and identifies disabled roaming as the next likely cause.This behavior is described as a more thorough, multi-step investigation.Source: §G.4 · pp. 43–44