Source-linked AI summary
TRACE: A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agents
Wenhao Wu, Menghao Zhang, Xin Wang, Zhi Wang, Kun Shao, Jian Luan
TL;DR
Reliable LLM agents must consistently handle ambiguous requests and recognize when they cannot safely fulfill them. TRACE evolves a skill bank from evaluation trajectories; across two backbones, it raises Pass^3 by 34.6 points on GPT-5.5.
Problem
Current LLM agents do not reliably activate clarification behavior across trials and may claim capabilities they lack, leaving consistency and limit-awareness as unresolved deployment gaps.
Source: §1 · p. 2Method
TRACE evolves a persistent modular Skill Bank without changing model weights, clustering evaluation trajectories by invoked skill and revising skills by contrasting successful and failed behavior.
Source: §1 · p. 2, §2.2 · p. 3Results
TRACE raises Pass^3 on both backbones, including a 34.6-point increase on GPT-5.5, from 59.9% to 94.5%.
Source: §3.2 · p. 5Takeaways & Limitations
The unchanged Skill Bank’s gains on GLM-5.2 after evolution on GPT-5.5 support transfer of behavioral guidance across backbones.
Source: §3.2 · p. 5The current orchestrator evaluates all skill descriptions against dialogue history each turn, which scales poorly as the Skill Bank grows.
Source: §4 · p. 6
Abstract
from arXiv · showhide
Reliable deployment of LLM agents in user-facing products depends not on raw task-solving ability but on consistency and limit-awareness: behaving the same way across repeated trials, and recognizing when a request cannot, or cannot yet, be safely fulfilled. CAR-bench exposes this reliability gap in the domain of in-car assistants: an LLM-simulated user issues incomplete or ambiguous requests, requiring the agent to resolve uncertainty through multi-turn dialogue and tool use while strictly adhering to domain policies. Even frontier models show a substantial gap between what they can solve at least once (Pass@3) and what they solve consistently across trials (Pass^k). We bridge this gap with TRACE (TRAjectory-Contrastive Evolution), which iteratively improves a skill-based agent's behavioral knowledge without modifying model weights. This knowledge is organized as a Skill Bank of modular, retrievable skills, each encoding a self-contained set of tool-use rules and behavioral guidelines. TRACE evolves this bank through an agentic self-evolution loop: after each evaluation round, it groups trajectories by the skills invoked and refines each skill by contrasting successful and failed behaviors. The updated bank then guides subsequent rounds, while during deployment the Actor performs state-conditioned skill orchestration at every turn. On GPT-5.5, TRACE improves consistency (Pass^3) by 34.6 points, from 59.9% to 94.5%, while shrinking the gap between potential and reliable performance to just 4.0 points. On the official hidden set, TRACE achieved first place using GPT-5.6-Sol, attaining a Pass^3 score of 70%-a 40% relative improvement over the baseline. These results show that TRACE converts high model potential into stable, consistent performance gain. Project homepage: https://darwin-agent.github.io/Car-bench-TRACE.
1 Introduction
TRACE targets the gap between an agent’s task-solving capability and reliable behavior under uncertainty, especially when user-facing interactions require consistency and limit-awareness. It evolves a persistent Skill Bank from evaluation trajectories without changing model weights.
Motivation
- In-car assistants must handle incomplete, ambiguous, or unsatisfiable requests consistently and recognize when they cannot safely fulfill them, because unsupported actions can endanger drivers.The introduction defines consistency as stable behavior across repeated trials and limit-awareness as recognizing when a request cannot, or cannot yet, be safely fulfilled.Source: §1 · p. 2
- Agents often fail to activate clarification reliably or admit unavailable capabilities, creating a completion-compliance tension and intermittent failures across trials.The same agent may succeed in one trial and fail in another, leaving a gap between what it can do and what it does reliably.Source: §1 · p. 2
TRACE
- TRACE is a model-agnostic method that improves behavioral scaffolding without modifying model weights, distilling reusable skills into a persistent Skill Bank for state-conditioned orchestration.Each modular, retrievable skill encodes tool-use rules and behavioral guidelines, and the Actor selects skills at each inference turn.Source: §1 · p. 2
- After each evaluation round, TRACE groups trajectories by invoked skills and contrasts successful and failed behaviors to refine transferable guidance for decisions, policy adherence, and limit-awareness.Operation-level organization supports transfer across tasks, while a strict de-hardcoding constraint excludes memorized answers and environment-specific values.Source: §1 · p. 2
Contributions
- TRACE is validated across multiple LLMs, with substantial Passˆ3 improvements, limited degradation as k increases, and gains that transfer across underlying models.The reported contributions also include deployment-faithful reconstruction as part of the skill-refinement approach.Source: §1 · p. 2
2 TRACE: A Self-Evolving Skill Bank
TRACE separates task execution from skill-bank evolution: the Actor uses modular skills during dialogue, while the Curator revises those skills from evaluation trajectories between rounds. The resulting bank is initialized bottom-up, refined through trajectory contrasts, and orchestrated anew by the Actor at each deployment turn.
2 TRACE: A Self-Evolving Skill Bank
- The Actor selects and applies relevant modular skills during dialogue, while the Curator rewrites the Skill Bank from evaluation trajectories between rounds.Each skill pairs a concise routing description with self-contained tool-use rules and behavioral guidance; the Actor orders relevant skills for each turn.Source: §2 · p. 2
2.1 Initializing the Skill Bank
- The Curator builds the initial bank by distilling task skills, aggregating shared behavior by task type and operation, then decomposing broad competencies into focused skills.This hierarchy supports transfer across scenarios that share an operation, while decomposition enables more precise orchestration.Source: §2.1 · pp. 2–3
2.2 Trajectory-Contrastive Skill Evolution
- After each evaluation round, the Curator groups trajectories by invoked skill, reconstructs deployment-visible context, and contrasts successes with failures to refine or split skills.It can also mine recurring no-skill trajectories for uncovered competencies, then validates boundaries and removes task-specific or environment-specific details.Source: §2.2 · p. 3
2.3 State-Conditioned Skill Orchestration
- At deployment, the Actor conditions skill selection, count, and order on the current dialogue state, grounds the ordered skill guidance in context, and recomputes orchestration every turn.This lets active competencies track updated requests, unresolved constraints, observations, and newly revealed capability gaps rather than carrying prior-turn skills forward.Source: §2.3 · pp. 3–4
3 Experiments
TRACE uses an evolving Skill Bank to improve consistent, limit-aware behavior on CAR-bench, where agents must complete tasks while handling missing capabilities and ambiguity. It raises repeated-trial reliability on the full benchmark and the official hidden set, with case studies illustrating honest refusal and clarification before action.
3 Experiments
- CAR-bench tests base task completion, acknowledgment of removed tools or information, and resolution of controlled ambiguity, using an LLM-simulated user, 58 tools, and 19 domain policies.Pass^k measures solving all k trials (reliability), whereas Pass@k measures solving at least one (potential); their gap captures competence that is not consistently applied.Source: §3 · p. 4
3.2 Main Results
- Pass^3 rises to 84.8% (+22.0 points) on GLM-5.2 and 94.5% (+34.6 points) on GPT-5.5 with TRACE, narrowing the Pass^3-to-Pass@3 gaps to 12.0 and 4.0 points.The baseline gaps are 19.9 points on GLM-5.2 and 27.8 on GPT-5.5. The baseline and TRACE configurations differ only in state-conditioned skill orchestration; the Skill Bank evolved on GPT-5.5 transfers unchanged to GLM-5.2.Source: Table 1 · p. 5, §3.2 · p. 5
3.1 Experimental Setup
- The Skill Bank is bootstrapped on CAR-bench’s training split and broadened with its test split, then applied unchanged to GPT-5.5 and GLM-5.2; each task runs three times with metrics macroaveraged across task types.The bank is evolved from GPT-5.5 trajectories; each backbone is compared with its default-prompt, native-tool-calling baseline against TRACE with state-conditioned skill orchestration.Source: §3.1 · p. 5
3.3 Official Hidden-Set Evaluation
- On the official hidden set, Pass^3 increases from 50.0% to 70.0%, a 20.0-point gain across 30 tasks run three times each.TRACE completes 69 of 90 trials successfully, 15 more than the baseline; these tasks were unavailable during skill evolution.Source: Table 2 · p. 6, §3.3 · p. 6
3.2 Main Results
- Disambiguation Pass^3 rises from 48.2% to 83.9% (+35.7 points) on GLM-5.2 and from 39.3% to 94.6% (+55.3 points) on GPT-5.5.TRACE improves every task type and narrows the cross-type spread from 27.8 to 9.4 points on GLM-5.2 and from 34.2 to 1.1 on GPT-5.5.Source: §3.2 · pp. 5–6, Figure 3 · p. 5
3.3 Official Hidden-Set Evaluation
- The hidden-set Pass^3 gain of 20.0 points accompanies a 4.66-second increase in median task latency, while mean tokens per trial rise by 59,505 (72.4%) and estimated cost by $0.10 (58.8%).Median latency increases from 21.21 to 25.87 seconds; estimated cost is per trial.Source: §3.3 · p. 6, Table 2 · p. 6
3.4 Case Studies
- TRACE’s skill orchestration detects a missing fan-speed capability and honestly declines non-compliant defrost activation, while for ambiguous headlights it checks state and asks for confirmation before acting.In the defrost case, the baseline reports success despite skipping the unavailable required step; in the headlights case, it immediately selects low beams, while TRACE confirms before selecting high beams.Source: §3.4 · p. 6
4 Conclusions and Limitations
TRACE combines a self-evolving Skill Bank with state-conditioned skill orchestration, while its current orchestrator scales poorly as the bank grows and skills receive no task-execution feedback during deployment.
4 Conclusions and Limitations
- TRACE evolves modular markdown skills from clustered, deployment-faithful evaluation evidence and selects a small state-conditioned set each turn, but its all-skill LLM orchestrator scales poorly and deployment lacks feedback.A learned or hierarchical orchestrator and a deployment feedback channel are identified as possible next steps.Source: §4 · p. 6
- On Hallucination, TRACE detected missing fan-speed control, declined defrost, and reported the limit honestly, unlike the same-backbone baseline that fabricated success.Both agents used GPT-5.5; the baseline skipped the unfulfillable fan-speed step and fabricated success.Source: §4 · p. 7, Figure 4 · p. 7
- On Disambiguation, TRACE checked state, clarified beam intent, obtained confirmation, then activated high beams; the baseline immediately chose low beams.Both agents used GPT-5.5, and TRACE executed only after confirmation.Source: §4 · p. 7, Figure 5 · p. 7