PaperlayerView exact revision on arXiv

Source-linked AI summary

SEAL: Synergistic Co-Evolution of Agents and Learning Environments

Yihao Hu, Zhihao Wen, Xiujin Liu, Pan Wang, Xin Zhang, Wei Wu

arXiv:2605.24426v1cs.CL

TL;DR

Current self-evolving tool-use agents often adapt the policy or training environment alone, despite changing capabilities and revealed failures. SEAL jointly evolves the training interface and policy using verifier-grounded diagnoses, improving low-resource learning across three backbones by up to +26.25 average points with 400 samples and transferring positively to held-out settings.

  • Problem

    Current self-evolving agents often adapt only the policy or environment, limiting learning signals in multi-turn tool-use settings with strict constraints and partial failures.

  • Method

    SEAL uses verifier-grounded failure diagnoses to jointly adapt the training-time interface and reweight policy optimization while preserving benchmark protocols.

  • Results

    +26.25 average-point gains: SEAL improves low-resource multi-turn tool-use learning across three backbones with only 400 training samples and positive transfer to held-out settings.

  • Takeaways & Limitations

    Adapting the training-time interface to revealed failures can make tool-use learning more targeted, sample-efficient, and robust to structured multi-turn errors.

  • Takeaways & Limitations

    SEAL relies on executable environments with tool schemas, execution traces, and verifier feedback, while cross-domain generalization remains challenging.

Abstract

from arXiv · show

Large Language Model (LLM) agents are increasingly improved through interaction, yet most self-evolution methods adapt either the policy or the learning environment in isolation. We identify this structural gap as \emph{Agent-Environment Misalignment}: the agent's capability frontier changes during training, while the environment that provides supervision remains static or only weakly coupled to the agent's revealed failures. We propose SEAL, a closed-loop co-evolution framework for interactive tool-use agents. SEAL collects on-policy trajectories under executable verification, diagnoses failed rollouts into turn-level failure labels, and uses these diagnoses as a shared signal for both environment-side adaptation and model-side policy optimization. The environment evolves its training-time learning interface by exposing clearer tool affordance cues, constraint information, and recovery-oriented feedback, while the policy is updated with diagnosis-guided advantage reweighting. Extensive experiments across in-distribution and out-of-distribution multi-turn tool-use evaluations show that SEAL improves low-resource agent learning: with only 400 training samples, it yields +8.25 to +26.25 average-point gains across three backbones and exhibits positive out-of-distribution transfer. These results demonstrate the value of jointly adapting the learner and its training-time learning substrate for robust self-improving LLM agents.

1 Introduction

The introduction identifies Agent-Environment Misalignment as a structural weakness in self-evolving tool-use agents and presents SEAL as a closed-loop framework that jointly adapts the training interface and policy. SEAL uses verifier-grounded failure diagnoses to improve low-resource multi-turn tool-use learning.

  • Problem: One-sided model-centric adaptation optimizes policies against largely fixed environments, making learning signals dependent on the current rollout distribution.This is especially limiting in long-horizon interactive settings with sparse rewards.
  • Problem: Agent-Environment Misalignment arises when an agent’s evolving capability frontier is not tracked by a static or weakly coupled training-time learning environment.This mismatch produces signals that are too static, weakly targeted, or insufficiently informative.
  • Motivation: Generic execution errors provide weak diagnostic supervision because they do not distinguish missing prerequisites, invalid argument types, or failed recovery.For example, an airport-code requirement may be surfaced only as a generic failure when the agent supplies a city name.
  • Approach: SEAL uses verifier-grounded failure diagnoses as a shared signal to evolve the training interface and guide policy optimization.Its environment-side adaptations include schema cues, constraint information, and recovery-oriented feedback, while policy updates are reweighted by diagnostic utility.
  • Results: +26.25 average-point gains are reported across three backbones for low-resource multi-turn tool-use learning.The introduction frames this result as evidence that jointly evolving the learning interface and policy benefits agent improvement.

2 Related Work

Related work largely improves either the agent through interaction or the training environment through curriculum and interface adaptation. SEAL combines these directions through failure-conditioned environment adaptation within realistic interactive agent-learning settings.

  • Model-centric agent improvement: Model-centric methods refine policies, prompts, memories, or skill libraries through repeated interaction, while leaving the training-time learning environment largely unchanged.These methods include recursive skill learning, self-consolidation, reflective prompt adaptation, memory-based improvement, and reinforcement learning from interaction feedback.
  • Training-environment adaptation: Curriculum learning, synthetic instruction generation, task evolution, and tool or skill construction reshape training distributions or interfaces, typically through task diversity, difficulty, or coverage.SEAL instead performs failure-conditioned environment adaptation using verifier-grounded diagnoses to determine environment changes.
  • Interactive benchmarks and co-evolution: Interactive benchmarks expose multi-turn dependencies, execution constraints, sparse rewards, and recovery dynamics across tool use, navigation, operating-system control, embodiment, and software engineering.Related work also studies environment design and agent–environment co-evolution, positioning SEAL within this broader interactive-learning perspective.

3 Methodology

SEAL co-evolves tool-use policies with a training-time learning interface by converting executable interaction failures into structured diagnoses that guide both interface adaptation and policy optimization. It preserves benchmark semantics, rewards, labels, and verification while improving the usefulness of training feedback.

  • Interface evolution: The framework evolves only the training-time learning interface, exposing tool affordance cues, constraint information, recovery feedback, and capability-specific hints while preserving tool semantics and benchmark fairness.Adaptation changes how existing environment information is presented, not the executable backend, verifier, tool signatures, outputs, or task labels.
  • Diagnostic interface: SEAL turns failed interactions into verifier-grounded, turn-level diagnoses that supplement sparse binary rewards without changing the benchmark reward or evaluation protocol.Diagnoses use parser checks, tool-schema validation, execution errors, observable state transitions, and verifier comparisons.
  • Policy optimization: Diagnosis-guided advantage reweighting assigns greater optimization pressure to failures with concrete executable evidence and clear repair directions, while preserving verifier-induced advantage signs and rankings.Invalid tool calls and argument mismatches receive higher utility than more ambiguous response mismatches; clipping limits rare or noisy updates.
  • Closed-loop training: SEAL alternates rollout collection, executable verification, failure diagnosis, interface evolution, and diagnosis-weighted GRPO in a closed co-evolution loop.The agent’s revealed capability gaps update the interface, and the policy internalizes the resulting feedback through optimization.

4 Experiments

Experiments show that SEAL improves low-resource multi-turn tool-use learning across three backbones, transfers to held-out benchmarks, and benefits from combining diagnosis-guided policy optimization with adaptive training interfaces.

  • Experimental setup: SEAL is evaluated with 400 BFCL V3 training examples across four categories, using three backbones and controlled Vanilla RL comparisons, plus held-out BFCL V4 and τ2-bench tests.The BFCL V3 benchmark contains 800 multi-turn examples, with 400 used for training and 400 for held-in evaluation.
  • In-distribution results: +8.25, +26.25, and +14.75 points are the average-score gains for Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and ToolACE-2-Llama-3.1-8B over their original checkpoints.Against Vanilla RL under matched training conditions, SEAL outperforms by +4.75, +9.50, and +8.25 average points across the three backbones.
  • In-distribution results: For Qwen2.5-7B-Instruct, SEAL raises Missing Functions from 14.00% to 36.00% and Missing Parameters from 10.00% to 34.00%.These structured tool-use categories align with failures involving missing tool use and invalid arguments, while Long Context gains suggest improved state tracking and recovery.
  • Out-of-distribution transfer: SEAL improves held-out performance: for Qwen2.5-3B-Instruct, BFCL V4 rises from 4.69% to 8.63% and τ2-bench from 9.64% to 11.76%.The results suggest transfer of argument grounding, error recovery, and multi-turn state management, although absolute OOD scores remain modest.
  • Ablations and training dynamics: Removing diagnosis-guided reweighting causes a larger drop than removing environment-side adaptation, while environment-side adaptation contributes +4.50 points over its ablated variant.Full SEAL outperforms both single-sided variants, indicating complementary contributions from adaptive feedback and diagnosis-guided optimization.
  • Ablations and training dynamics: SEAL improves more rapidly and reaches higher validation accuracy on most subsets, especially Missing Functions and Missing Parameters, with a widening post-early-phase gap over Vanilla RL.The smoother, higher trajectory is consistent with updating training cues as dominant policy failures change instead of retaining a sparse terminal signal.

5 Conclusion and Limitations · Appendix

SEAL co-evolves tool-use agents and their training-time learning environments through verifier-grounded failure diagnosis. The conclusion emphasizes targeted, sample-efficient, and robust learning while identifying supervision, adaptation-scope, and diagnostic-weight limitations.

  • 5 Conclusion and Limitations: SEAL is a closed-loop framework for co-evolving tool-use agents and their training-time learning environments.It uses verifier-grounded failure diagnosis as a shared signal for interface evolution and policy optimization.
  • 5 Conclusion and Limitations: Verifier-grounded failure diagnosis addresses Agent-Environment Misalignment while preserving tool semantics, task labels, and the evaluation verifier.The shared diagnosis signal drives both learning-interface evolution and policy optimization.
  • 5 Conclusion and Limitations: SEAL’s experiments show consistent gains in low-resource learning across in-distribution and out-of-distribution multi-turn tool-use benchmarks.The supplied conclusion passage reports these benchmark findings without specifying the corresponding numerical values.
  • 5 Conclusion and Limitations: Adapting the training-time interface to revealed failures makes learning more targeted, sample-efficient, and robust to structured multi-turn errors.The conclusion contrasts this approach with optimizing only the policy against a fixed environment.
  • 5 Conclusion and Limitations: SEAL relies on executable environments with tool schemas, execution traces, and verifier feedback, limiting direct application to more open-ended domains.Such domains may require richer diagnosis mechanisms and weaker forms of supervision.
  • 5 Conclusion and Limitations: Conservative environment evolution modifies the training-time interface rather than tool semantics or evaluation rules, preserving fairness but limiting adaptation scope.This design deliberately avoids changing the underlying tools or evaluation criteria.
  • 5 Conclusion and Limitations: Diagnostic utility weights are fixed across tasks and backbones, creating a limitation for adapting diagnosis to varied settings.The passage identifies fixed utility weighting as an unresolved limitation but provides no numerical comparison.
  • 5 Conclusion and Limitations: Future work will extend diagnosis to less structured environments, learn adaptive utility weights, and test scaling to broader tool ecosystems and longer-horizon workflows.These directions target the framework’s supervision, weighting, and scalability limitations.

A Benchmark and Evaluation Details

The evaluation combines BFCL V3 in-distribution testing with BFCL V4 and τ2-bench out-of-distribution testbeds. All methods use a shared 400-example BFCL V3 training split, and evaluation uses original benchmark protocols with success rate as the primary metric.

  • In-distribution benchmark: BFCL V3 evaluates multi-turn executable tool use across 800 examples in Base, Missing Functions, Missing Parameters, and Long Context categories.It is the in-distribution evaluation benchmark.
  • Data splits: 400 training examples are sampled evenly across BFCL V3 categories, with the remaining 400 examples reserved for held-in evaluation.SEAL and Vanilla RL use identical training and held-in test splits.
  • Out-of-distribution benchmarks: BFCL V4 Web Search and Memory tracks test out-of-distribution retrieval, synthesis, and memory operations using tool schemas and interaction patterns differing from BFCL V3.SEAL is trained only on the BFCL V3 training split.
  • Out-of-distribution benchmarks: τ2-bench evaluates multi-turn tool use in Retail, Airline, and Telecom customer-service domains with domain policies, constraints, and persistent task state.Its different tool semantics and interaction patterns provide a complementary OOD generalization testbed.
  • Protocol and metric: Evaluation uses original benchmark environments and verifiers, disables SEAL-specific feedback and hints, and reports success rate as the primary metric.Category scores use corresponding subsets, while average scores are arithmetic means over reported categories.

B SEAL Training Algorithm

SEAL alternates verifier-grounded policy rollouts, trajectory diagnosis, interface evolution, and policy optimization in a closed training loop. The training-time interface wrapper is removed for final evaluation in the original environment with the unchanged verifier.

  • Training loop: Each round constructs a wrapped training environment, samples prompt batches, and collects trajectories from the current policy for verifier-based analysis.The wrapper is formed as ℰ_Ωr ← Ω_r(ℰ), and rollouts are generated from π_θr in ℰ_Ωr.
  • Training loop: SEAL computes verifier rewards and turn-level diagnoses for each trajectory, stores them jointly, and aggregates diagnoses into a round-level failure profile.The algorithm stores (x, τ_j, r_j, Z(τ_j)) and computes C_r ← Agg({Z(τ_j)}).
  • Policy optimization: The policy computes group-relative advantages, applies diagnostic weights, and updates with GRPO plus KL regularization.Weighted advantages are set as Ã_j ← w_j A_j before updating π_θr → π_θr+1.
  • Co-evolution and evaluation: The interface evolves from Ω_r to Ω_r+1 using the aggregated failure profile, while final evaluation discards the wrapper and uses the original environment and unchanged verifier.Interface updates follow Ω_r+1 ← U(Ω_r, C_r); evaluation uses π_θR in ℰ with verifier 𝒱.

C Experimental Details … E Qualitative Case Studies

SEAL’s experiments use controlled Vanilla RL comparisons, diagnosis-based training adaptations, and reproducible interface configurations. Qualitative cases illustrate improvements across four multi-turn tool-use capability types while preserving identical test-time conditions.

  • C Experimental Details: SEAL and Vanilla RL share the base model, 400 BFCL V3 training samples, rollout budget, optimizer, decoding configuration, verifier, and evaluation environment.The controlled protocol isolates SEAL’s verifier-grounded diagnoses, training-time interface updates, and diagnostic utility weights.
  • C.2 Diagnosis Label Priority: SEAL prioritizes directly executable failures over downstream verifier failures, assigning argument_mismatch instead of response_mismatch when an invalid earlier tool argument caused final verification failure.Trajectory-level diagnostic weights are clipped to [𝑤min, 𝑤max] = [0.5, 2.0].
  • C.3 Controlled Comparison Protocol: 400 BFCL V3 training samples are used in the controlled Vanilla RL–SEAL comparison, with both methods evaluated through the original BFCL V3 multi-turn environment and official verifier.Training excludes ground-truth tool trajectories, hidden parameter values, reference intermediate actions, and final answers.
  • C.3 Controlled Comparison Protocol: SEAL’s evolved interface is removed during evaluation, so both methods interact with the original benchmark interface under identical test-time conditions.Performance therefore reflects policy behavior rather than access to additional test-time information.
  • C.4 Logging and Reproducibility: Each run logs seeds, checkpoints, code versions, configurations, rollout and decoding parameters, validation scores, failure profiles, interface states, diagnostic distributions, and final outputs.These logs support attribution of performance changes to SEAL components rather than differences in training or evaluation protocols.
  • D Prompt and Interface Templates: The base tool-use prompt is shared by Vanilla RL and SEAL, while SEAL changes only the training-time environment interface and disables all SEAL-specific augmentations during evaluation.The base prompt exposes tool schemas and specifies JSON tool-call formatting inside <tool_call> tags.
  • D.2 Schema Observation Annotation: Observation-lite annotations make required arguments and enum constraints explicit without changing tool APIs, signatures, outputs, or verifier behavior.They expose public schema information but not correct tool choices, hidden parameters, reference trajectories, or final answers.
  • D Prompt and Interface Templates: SEAL can convert parser and execution errors into non-solution-revealing recovery hints, while diagnostic evolution injects capability-level guidelines derived from recurring failure patterns.The documented optional components address formatting, missing or extra arguments, argument types, spurious or missing calls, state mismatches, and response-format errors.

E.1 File-System State Tracking

In the file-system tracking case, the base model failed to maintain state and earned reward = 0.0, whereas Base model + SEAL tracked directory contents and tool constraints to complete the workflow with reward = 1.0.

  • Task: The task required comparing report files, moving and renaming a note within an archive, retrieving a support ticket, and resolving it.The central challenge was maintaining the current directory state while respecting file-system tool constraints.
  • Base model: reward = 0.0 for the Base model, which initially failed to locate the relevant files and continued issuing invalid path-style mv arguments.The failure analysis attributes this behavior to not converting execution errors into a revised, state-aware plan.
  • Base model + SEAL: The SEAL-trained model retrieved ticket 987654 and resolved it successfully with the provided manual-troubleshooting resolution.The tool returned: Ticket 987654 has been resolved successfully.
  • Base model + SEAL: reward = 1.0 for Base model + SEAL, which inferred actual filenames from directory contents and compared report_draft.txt with report_final.txt.The model used ls after failed searches and then issued diff on the discovered filenames.
  • Base model + SEAL: SEAL moved temp_notes.txt into archives, changed into /archives, and renamed the file to notes_2024.txt.This sequence respected the tool’s local-directory destination constraint.

E.2 Recovering from API Parameter Mismatch · E.3 Cross-Domain Tool Composition

SEAL improves multi-turn tool use by recovering from invalid parameters and preserving user-specified entities across API workflows. In the cases shown, it resolves missing intermediate information, applies required conversions, and updates existing cross-domain artifacts correctly.

  • E.2 Recovering from API Parameter Mismatch: Base model receives reward = 0.0 after treating guessed airport codes as final and setting the GBP amount without conversion.It reports no direct route after using RVM and STB, then sets 15400 GBP directly as the budget limit.
  • E.2 Recovering from API Parameter Mismatch: SEAL recovers from the route error by resolving Rivermist and Stonebrook to RMS and SBK before querying the flight cost.The corrected call returns a travel cost list containing 220.0.
  • E.3 Cross-Domain Tool Composition: Base model receives reward = 0.0 after substituting Apple for Omega Industries and posting a new tweet instead of updating the existing one.The initial OMGA lookup fails, after which the model uses AAPL and later reposts content with both mentions.
  • E.3 Cross-Domain Tool Composition: SEAL treats the failed OMGA lookup as recoverable evidence, resolves Omega Industries to OMEG, and retrieves price 457.23.It uses get_symbol_by_name before repeating get_stock_info with the resolved symbol.
  • E.3 Cross-Domain Tool Composition: SEAL receives reward = 1.0 by posting the correct Omega Industries content with #TechBoom and @industryexperts, then adding @technewsworld to the existing tweet.The dedicated mention tool reports that users were mentioned successfully.

E.4 Vehicle-Control State Maintenance

SEAL resolves a long-context vehicle-control failure by executing the required tank refill and preserving that state for a later engine check. On BFCL multi_turn_long_context_88, reward rises from 0.0 for the base model to 1.0 with Base model + SEAL.

  • Expected behavior: The complete expected sequence converts 13.2 gallons to approximately 49.97 liters, fills to 50.0 gallons, maintains vehicle state, and posts the road-trip tweet with three specified hashtags.The hashtags are #RoadTrip, #Adventure, and #Exploring.
  • Base model failure: The base model earns reward = 0.0 because it computes the remaining capacity but never executes fillFuelTank, leaving the later fuelLevel at 13.2.The missing state update causes the subsequent engine check to report the original fuel level.
  • SEAL success: SEAL-trained execution earns reward = 1.0 by calling fillFuelTank with fuelAmount = 36.8 before locking all doors and engaging the parking brake.The tool responses confirm fuelLevel = 50.0, lockStatus = locked, remainingUnlockedDoors = 0, and parkingBrakeStatus = engaged.
  • SEAL success: The later engine sequence recovers from the brake requirement, starts the engine, and observes fuelLevel = 50.0, confirming preservation of the long-range state dependency.The model presses the brake pedal before retrying startEngine in START mode.

E.5 OOD Generalization on 𝜏2-Bench Mobile Data Troubleshooting

On τ2-Bench Mobile Data Troubleshooting, the vanilla Qwen2.5-3B agent partially fixes roaming but fails to resolve the task, while SEAL continues diagnosis through Data Saver and reaches Excellent speed. The SEAL trajectory improves speed from 55.00 Mbps (Good) to 275.00 Mbps (Excellent).

  • Vanilla Qwen2.5-3B achieves reward = 0.0 on τ2-bench Telecom Mobile Data Troubleshooting.
  • The baseline enables data roaming but ignores the remaining Data Saver restriction, drifts toward unrelated steps, and escalates without resolving the task.After roaming is enabled, the speed test remains Good while Data Saver is still shown in the status bar.
  • SEAL continues diagnosis after roaming produces only 55.00 Mbps (Good), checks data restrictions, and identifies that Data Saver mode is ON.
  • 275.00 Mbps (Excellent) is achieved after SEAL turns Data Saver mode OFF, resolving the mobile-data troubleshooting task.The user confirms that the speed is now excellent and stops the interaction.
Loading 2605.24426v1…