Source-linked AI summary
TRACE: A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agents
Wenhao Wu, Menghao Zhang, Xin Wang, Zhi Wang, Kun Shao, Jian Luan
TL;DR
LLM agents need consistency and limit-awareness when handling ambiguous or unsatisfiable in-car requests, yet current systems often fail intermittently or claim unsupported success. TRACE evolves a modular Skill Bank from contrasting successful and failed trajectories, then orchestrates skills conditionally at each deployment turn. On GPT-5.5, it raises Pass^3 from 59.9% to 94.5% and narrows the Pass^3-to-Pass@3 gap to 4.0 points.
Problem
Current LLM agents struggle to consistently resolve ambiguity and acknowledge unavailable capabilities, creating a reliability gap between potential and repeated-trial performance.
Method
TRACE evolves a persistent modular Skill Bank by grouping evaluation trajectories by invoked skills and contrasting successful and failed behaviors, without modifying model weights.
Results
34.6 points: TRACE raises GPT-5.5 Pass^3 from 59.9% to 94.5% and shrinks the Pass^3-to-Pass@3 gap to 4.0 points.
Takeaways & Limitations
TRACE converts existing model competence into more consistently reproducible, limit-aware behavior across repeated trials and underlying backbones.
Takeaways & Limitations
The current LLM-based orchestrator evaluates all skill descriptions at every turn, which suits the small bank used here but scales poorly as the bank grows.
Abstract
from arXiv · showhide
Reliable deployment of LLM agents in user-facing products depends not on raw task-solving ability but on consistency and limit-awareness: behaving the same way across repeated trials, and recognizing when a request cannot, or cannot yet, be safely fulfilled. CAR-bench exposes this reliability gap in the domain of in-car assistants: an LLM-simulated user issues incomplete or ambiguous requests, requiring the agent to resolve uncertainty through multi-turn dialogue and tool use while strictly adhering to domain policies. Even frontier models show a substantial gap between what they can solve at least once (Pass@3) and what they solve consistently across trials (Pass^k). We bridge this gap with TRACE (TRAjectory-Contrastive Evolution), which iteratively improves a skill-based agent's behavioral knowledge without modifying model weights. This knowledge is organized as a Skill Bank of modular, retrievable skills, each encoding a self-contained set of tool-use rules and behavioral guidelines. TRACE evolves this bank through an agentic self-evolution loop: after each evaluation round, it groups trajectories by the skills invoked and refines each skill by contrasting successful and failed behaviors. The updated bank then guides subsequent rounds, while during deployment the Actor performs state-conditioned skill orchestration at every turn. On GPT-5.5, TRACE improves consistency (Pass^3) by 34.6 points, from 59.9% to 94.5%, while shrinking the gap between potential and reliable performance to just 4.0 points. On the official hidden set, TRACE achieved first place using GPT-5.6-Sol, attaining a Pass^3 score of 70%-a 40% relative improvement over the baseline. These results show that TRACE converts high model potential into stable, consistent performance gain. Project homepage: https://darwin-agent.github.io/Car-bench-TRACE.
A Technical Report on the CAR-bench Challenge
CAR-bench targets reliable in-car assistance, where agents must handle capability boundaries and repeated-rollout consistency. TRACE uses state-conditioned skill orchestration to narrow the gap between potential and reliable performance.
- CAR-bench evaluates whether LLM agents recognize capability boundaries and behave consistently across repeated rollouts.
1 Introduction
Reliable in-car assistants must resolve incomplete or unsatisfiable requests while maintaining consistency and limit-awareness under domain policies. TRACE addresses this gap by evolving reusable behavioral knowledge without modifying model weights.
- User-facing agents need consistency across repeated trials and limit-awareness when requests cannot safely be fulfilled.These properties are safety-critical for in-car assistants because unsupported actions can distract or endanger drivers.
- Intermittent failures create a gap between what an agent can do and what it does reliably.
- TRACE evolves a persistent Skill Bank of modular, retrievable skills encoding tool-use rules and behavioral guidelines without modifying model weights.The Actor performs state-conditioned skill orchestration at each inference turn.
- TRACE refines skills by contrasting successful and failed evaluation trajectories, converting unreliable decisions into reusable guidance for limit-awareness and policy adherence.
- TRACE validates the evolved Skill Bank across multiple LLMs, reporting substantial Pass^3 improvements with limited degradation as k increases and transfer across underlying models.
2 TRACE: A Self-Evolving Skill Bank
TRACE separates task execution from skill optimization: the Actor orchestrates relevant competencies during dialogue, while the Curator initializes and evolves a modular Skill Bank from evaluation trajectories. Deployment repeatedly selects and grounds skills according to the current dialogue state.
- TRACE architecture: The Actor executes tasks and tools, while the Curator rewrites behavioral knowledge from evaluation evidence for later runs.The Skill Bank stores modular competencies as descriptions paired with self-contained tool-use rules and behavioral guidelines.
- Skill Bank initialization: The Curator initializes the bank through task-level distillation, type-level aggregation, operation-level abstraction, and skill decomposition.This pipeline is summarized as Btask → Btype → Bop → B(0).
- Skill Bank initialization: Hierarchical abstraction transfers knowledge across scenarios sharing an operation, while decomposition enables focused activation.
- Trajectory-contrastive evolution: After each evaluation round, TRACE groups trajectories by invoked skills, reconstructs deployment-visible evidence, and rewrites skills by contrasting success and failure.Unrecognized or skill-free trajectories are set aside for possible skill mining or revision.
- State-conditioned orchestration: During deployment, the Actor jointly chooses relevant skills, activation count, and composition order from the current dialogue state.The selected skill bodies are combined with dialogue history, tool rules, behavioral guidelines, and procedures to form the generation context.
- State-conditioned orchestration: Skill orchestration is recomputed after every turn, allowing active competencies to track newly underspecified requests or missing capabilities.The bodies injected on one turn are not carried over automatically to the next.
3 Experiments
TRACE is evaluated on CAR-bench across task types, backbones, and an unseen hidden set using repeated-trial reliability and potential metrics. It substantially improves consistency, transfers across backbones and unseen tasks, and concentrates gains on difficult disambiguation behaviors, with added latency and resource costs.
- Benchmark and Metrics: CAR-bench tests in-car assistants with 58 tools and 19 domain policies across Base, Hallucination, and Disambiguation tasks.Each task is run repeatedly to measure consistency rather than one-shot success alone.
- Benchmark and Metrics: Pass^k measures tasks solved in all k trials, whereas Pass@k measures tasks solved in at least one trial, separating reliability from potential.The Pass@k−Pass^k gap captures competence that is not applied consistently.
- Overall Results: 34.6 points: GPT-5.5 Pass^3 rises from 59.9% to 94.5%, shrinking the Pass^3-to-Pass@3 gap from 27.8 to 4.0 points.On GLM-5.2, Pass^3 rises to 84.8% (+22.0), with the gap shrinking to 12.0 points.
- Transfer: The Skill Bank transfers across backbones: evolved solely on GPT-5.5 trajectories, it produces comparable gains when applied unchanged to GLM-5.2.The baseline and TRACE differ only in state-conditioned skill orchestration, isolating the contribution of the evolved competencies.
- Task-Type Results: TRACE improves every task type, with GPT-5.5 Disambiguation rising from 39.3% to 94.6% (+55.3).The cross-type spread narrows from 34.2 to 1.1 points on GPT-5.5 and from 27.8 to 9.4 on GLM-5.2.
- Hidden-Set Results: On the GPT-5.6-Sol hidden set, Pass^3 increases from 50.0% to 70.0% (+20.0 points), while median latency rises from 21.21 to 25.87 seconds.Mean tokens per trial increase by 59,505 (72.4%), and estimated cost rises by $0.10 per trial (58.8%).
4 Conclusions and Limitations
TRACE improves behavioral reliability by detecting unavailable capabilities and resolving ambiguous requests before acting. Its current skill orchestration remains limited by exhaustive LLM evaluation over the Skill Bank and the lack of deployment-time feedback.
- Limitations: The Actor evaluates all skill descriptions against dialogue history at each turn, an approach suited to the small bank but poorly scalable as the bank grows.A learned or hierarchical orchestrator is identified as a natural next step.
- Limitations: TRACE has no feedback channel from task execution back into the Skill Bank during deployment.Adding such a channel could enable mid-dialogue adaptation or correction.
- Limit-awareness: TRACE detects missing fan-speed control and declines to activate defrost instead of reporting fabricated compliance.The baseline omits the mandatory fan-speed step and claims defrost is active, while TRACE honestly reports the limitation.
- Internal disambiguation: TRACE checks light state, clarifies the intended beam type, obtains confirmation, and then activates high beams.The baseline interprets the ambiguous request as low beams without checking state or clarifying intent.