PaperlayerView exact revision on arXiv

Source-linked AI summary

TRIAGE: Three-level Routing and Intelligent Agent Guidance for Efficient Execution

Ruocan Wei

arXiv:2609.01428v1cs.LG

TL;DR

TRIAGE addresses repeated from-scratch reasoning on similar queries by reusing historical execution trajectories as Skills and routing queries among three execution levels. It reports token reduction on its query set and with real threshold routing; trajectory errors may propagate without verification mechanisms.

  • Problem

    ReAct starts reasoning from scratch for each query, while existing skill systems require fixed-schema modules that have weak generalization and high maintenance costs.

    Source: §1.1 · p. 2, §2.3 · p. 5

  • Method

    TRIAGE uses TaaS to abstract historical execution trajectories into reusable Skills and applies three-level routing, with no pre-training or annotation required.

    Source: §3.2 · p. 7, §4.4.1 · p. 18, §6 · p. 25

  • Results

    62.3% token reduction (199,782 →75,238) on 1,007 queries; 76.3% token reduction with real threshold routing.

    Source: §6 · pp. 25–26

  • Takeaways & Limitations

    TaaS makes historical execution trajectories reusable as Skills, and the paper describes this trajectory store as a byproduct of normal operation.

    Source: §3.2 · p. 7, §6 · p. 25

    Trajectory errors may propagate without verification mechanisms.

    Source: §5.2 · p. 24

Abstract

from arXiv · show

Large Language Model (LLM) agents based on the ReAct paradigm have demonstrated remarkable capabilities in tool use and task execution. However, ReAct suffers from a fundamental efficiency problem: every query triggers a complete reasoning loop from scratch, and similar queries repeat identical steps without leveraging historical experience. We propose TRIAGE,a three-level routing framework that reduces token consumption by reusing historical execution trajectories. Its core innovation is TaaS (Trajectory-as-a-Skill), which abstracts historical execution trajectories into reusable skills, realizing 'experience as a service'. TRIAGE classifies queries into three levels: (1) Direct Reuse-identical queries, 0 tokens; (2) Skill Substitution-similar queries, 0 tokens via deterministic parameter substitution; (3) Full ReAct-novel queries, automatically stored for future reuse. In large-scale experiments on 1,007 security monitoring queries, TRIAGE achieves 62.3% token savings, with 56.0% of queries at Level 2 and 5.5% at Level 1, both executing at zero cost. Cross-domain validation on ToolBench (15 domains, 345 queries) achieves 76.3% token reduction, confirming the generalizability of semantic routing. An online learning experiment demonstrates cold-start-to-mature evolution: the L2 hit rate rises from 0% to 57% within the first 100 queries, and the average token cost drops from 198 to 74.7. We also propose an automatic Skill extraction mechanism that distills high-frequency trajectory patterns into deterministic Skills, creating a positive feedback loop of 'the more you use it, the more efficient it becomes'.

1 Introduction

TRIAGE addresses repeated ReAct reasoning by routing queries to reuse historical execution experience, including parameter-varying queries. Its introduction positions this approach against existing efficiency methods and reports results across security-monitoring and ToolBench queries.

1.1 Motivation

  • ReAct repeats the full reasoning process for similar queries, including schema inference, SQL generation, execution, and formatting, at the same token cost.A single loop averages 198 tokens; 1,007 queries total 199,782 tokens.Source: §1.1 · p. 2

1.2 Limitations of Existing Approaches

  • Existing approaches leave a gap: TRIAGE is presented as a complete, validated framework for reusing historical trajectories to reduce future similar queries’ token consumption.Prompt compression shortens input, speculative decoding accelerates generation, and iterative-refinement methods add token consumption.Source: §1.2 · p. 3

1.3 Our Approach: TRIAGE

  • TRIAGE treats historical execution trajectories as reusable skills and routes identical, parameter-varying, and novel queries to direct reuse, Skill substitution, or full ReAct.Levels 1 and 2 use zero tokens; Level 3 queries are automatically stored for future reuse.Source: §1.3 · p. 3

1.4 Contributions

  • TRIAGE’s contributions include automatic cold start without pre-training or annotation and automatic distillation of high-frequency trajectories into fine-grained deterministic Skills.The introduction also describes semantic-encoder retrieval using 384-dimensional embeddings.Source: §1.4 · p. 4
  • 76.3% token reduction across 15 ToolBench domains (345 queries) validates semantic routing beyond SQL, while the framework reports 62.3% token reduction.The introduction also reports a 61.5% L1+L2 reuse rate.Source: §1.4 · pp. 3–4
  • The introduction reports that routing strategy choice depends on the ratio of routing overhead to per-query execution cost, rather than being universal.The comparison is between threshold routing and LLM-based routing.Source: §1.4 · p. 4

2 Related Work

Related work distinguishes tool-capable LLM agents from methods aimed at token efficiency and experience reuse. The paper frames its approach against methods that do not remove repeated reasoning and skill systems requiring predefined modules.

2.1 LLM Agents and Tool Use

  • Toolformer [Schick et al., 2023], ToolBench/ToolLLM, and Gorilla [Patil et al., 2023] extend tool use but do not address execution efficiency.The section identifies execution efficiency as a separate concern from tool-use capability.Source: §2.1 · p. 4

2.2 Token Efficiency Optimization

  • Prompt compression, speculative decoding, and caching are described as leaving redundant reasoning for similar queries fundamentally uneliminated.The cited examples include LLMLingua, Medusa, and GPTCache.Source: §2.2 · p. 4

2.3 Experience Reuse and Skill Systems

  • Existing Function Calling skill systems require developers to predefine fixed-schema modules, whereas TRIAGE presents trajectories as dynamically discovered Skills adapted via LLM editing.The paper characterizes the predefined-module approach as having weak generalization and high maintenance costs.Source: §2.3 · p. 5

3.1 System Architecture

The architecture encodes queries, retrieves similar trajectories, and routes them through three execution levels. Its components include semantic encoding, threshold-based routing, trajectory retrieval, and Skill extraction.

3.1 System Architecture

  • The semantic encoder uses all-MiniLM-L6-v2 to encode queries as 384-dimensional vectors for semantic similarity retrieval.The architecture identifies it as a sentence-transformers encoder.Source: §3.1 · p. 5
  • The deterministic similarity-threshold router uses cosine similarity thresholds of L1 ≥0.98 and L2 ≥0.90 to classify queries into three levels.For L1 candidates below 1.0, normalization and parameter filtering resolve most cases at zero token cost; unresolved differences trigger an LLM judgment.Source: §3.1 · p. 5
  • The trajectory retriever uses cosine similarity and thresholds to select historical trajectories, while the Skill Extractor matches distilled Skills for deterministic parameter substitution.When no Skill matches, execution falls back to full ReAct.Source: §3.1 · p. 5
  • The architecture describes automatic Skill extraction from high-frequency trajectory patterns as part of its three-phase system.The figure caption identifies semantic retrieval as a phase.Source: §3.1 · p. 6, Figure 1 · p. 6

3.2 TaaS: Trajectory-as-a-Skill

TaaS turns historical execution trajectories into reusable Skills, rather than leaving them as passive records. Its store–retrieve–reuse–distill lifecycle connects accumulated experience to deterministic reuse and further Skill extraction.

3.2 TaaS: Trajectory-as-a-Skill

  • TaaS abstracts historical execution trajectories into reusable Skills, treating successfully executed trajectories as latent skills discoverable through use.This contrasts with systems that require fixed-schema modules to be predefined at design time.Source: §3.2 · p. 7
  • A trajectory is a complete execution record containing the user query, tool-invocation sequence, and final result, persisted in a database indexed by semantic embeddings.The cited tool sequence examples include schema inference, SQL generation, execution, and formatting.Source: §3.2 · p. 7
  • Every successfully executed Level 2 or Level 3 query is appended to the trajectory database, with no manual annotation, schema design, or pre-training required.FAISS can be integrated as a drop-in accelerated index for large-scale deployments.Source: §3.2 · p. 7
  • On a new query, TRIAGE embeds it, retrieves the top-k similar trajectories, and compares their similarity scores against routing thresholds.The passage describes retrieval as the bridge between stored experience and reuse.Source: §3.2 · p. 7
  • Level 1 returns an exact-match stored result, Level 2 performs deterministic Skill parameter substitution at zero tokens, and Level 3 falls back to full ReAct.The router dispatches according to similarity score.Source: §3.2 · pp. 7–8
  • High-frequency trajectory patterns are distilled into deterministic Skills that expose typed parameters for variable slots while preserving recurring query structure.Matching later queries can execute through parameter substitution without LLM inference.Source: §3.2 · p. 8
  • The store–retrieve–reuse–distill loop is described as self-reinforcing: more usage generates trajectories and retrieval hits, which yield reuse savings and further distillation.The paper contrasts this usage-dependent cycle with static optimizations whose efficiency remains constant regardless of usage volume.Source: §3.2 · p. 8

3.3 Three-level Routing Mechanism

TRIAGE routes queries among exact reuse, Skill-based substitution, and full ReAct according to similarity and whether differences can be resolved as parameters. Unresolved cases can receive an LLM judgment that updates parameter extraction rules for future queries.

3.3 Three-level Routing Mechanism

  • Level 1 returns a stored historical SQL query at zero token cost when the query is an exact match.The routing description identifies exact matches with cosine similarity of 1.0.Source: §3.3 · p. 8
  • The system first matches a query against distilled Skills in the global Skill registry; when matched, it extracts parameter values and uses deterministic parameter substitution to execute at 0 token consumption.Source: §3.3 · p. 8
  • When normalization cannot resolve a difference, an LLM classifies it as a parameter change, cosmetic rewording, or semantically distinct query.For parameter changes, it extracts the parameter name and value and registers dynamic regular-expression rules for later queries with the same pattern.Source: §3.3 · p. 9
  • The feedback rules let later queries with the same parameter pattern be handled at zero token cost by normalization.The parameter extraction system learns from LLM feedback.Source: §3.3 · p. 9
  • Level 3 sends queries with cosine similarity below 0.90 through full ReAct and stores each resulting trajectory for Skill extraction.This route handles structurally novel queries.Source: §3.3 · p. 9

3.4 Online Learning Dynamics

In the sequential online experiment, TRIAGE moves from a cold start through warm-up to a mature phase as trajectories accumulate. Reuse expands over time, lowering the share of queries requiring full ReAct.

3.4 Online Learning Dynamics

  • The experiment processes 1,007 queries sequentially over 30 simulated days, with recurring, parameter-varying, and novel queries interspersed to model a cold start and accumulating experience.Daily queries are clustered at day starts, parameter queries are interspersed, and novel queries arrive at random intervals.Source: §3.4 · pp. 9–10
  • Parameter-varying queries make up 57.2% of the workload, and examples such as different device types or time windows can have cosine similarities of 0.35–0.77.These queries are generated from 15 parameterized SQL templates with random parameter combinations.Source: §3.4 · p. 10
  • In queries 1–50, all queries initially use full ReAct at approximately 197 tokens, with the first recurring queries beginning to hit Level 1 by the phase’s end.Trajectories accumulate and Skill extraction begins after the first query completes.Source: §3.4 · pp. 10–11
  • During queries 50–300, the Level 3 ratio falls from 100% to below 10% as recurring queries use Level 1 and parameter-varying queries increasingly match Skills at Level 2.The warm-up phase follows the initial cold-start period.Source: §3.4 · p. 11
  • By queries 300–1,000, Level 1 covers recurring queries, Level 2 covers most parameter variations, and Level 3 handles genuinely novel queries, approximately one per day.Average token cost stabilizes at a small fraction of the baseline.Source: §3.4 · p. 11
  • As usage accumulates, TRIAGE extracts more Skills and consumes fewer tokens per query, forming a self-reinforcing cycle that contrasts with static optimizations whose gains do not vary with usage volume.The cycle is described as usage leading to trajectories, Skills, savings, and more usage.Source: §3.4 · p. 11

3.5 Automatic Skill Extraction

TRIAGE continuously extracts reusable Skills from stored trajectories through a general-purpose pattern pipeline and a SQL-specific implementation. As use generates trajectories and extractions, the system builds a reusable competency library and its token efficiency improves.

3.5 Automatic Skill Extraction

  • Automatic Skill extraction continuously turns recurring execution patterns into registered, deterministic Skills, bootstrapping an executable competency library from raw experience.The authors describe this as a self-improving efficiency loop that accelerates with usage.Source: §3.5 · p. 11

General-Purpose Extraction Framework

  • The extraction framework normalizes invocations, accumulates recurring patterns, induces named parameters, and registers templates as Skills for zero-cost execution.The framework is described as action-type-agnostic and uses structural fingerprints; concrete tool families supply a normalizer.Source: General-Purpose Extraction Framework · pp. 12–13

SQL-Specific Implementation

  • SQL implementation replaces literals, dates, and numeric constants with typed placeholders while preserving structural identifiers, then executes parameterized templates by direct substitution without the LLM.Parameter positions are found by comparing instances, and semantic names use a lightweight domain lexicon.Source: SQL-Specific Implementation · p. 13

Positive Feedback Loop

  • More usage generates trajectories, which prompt more extractions and enable more zero-token direct reuse, unlike static optimizations whose gains remain constant with usage volume.The paper characterizes TRIAGE’s efficiency as a function of experience.Source: Positive Feedback Loop · p. 13

4 Experiments

The experiments evaluate TRIAGE’s token-efficiency gains and routing across the security-monitoring benchmark and ToolBench, including threshold-based routing, ablations, and online learning as trajectories accumulate.

4.1 Experimental Setup

  • The evaluation dataset comprised 1,007 English queries: 947 structurally diverse queries generated from 15 parameterized SQL templates and 60 fixed recurring daily-report queries, ordered across a 30-day structure.Source: §4.1 · p. 14

4.2 Main Results

  • 62.3% token reduction: TRIAGE used 75,238 tokens versus 199,782 for the baseline across 1,007 security-monitoring queries.Level-1 and Level-2 reuse handled 61.5% of queries at zero tokens: 55 direct-reuse queries and 564 Skill-substitution queries.Source: Table 1 · p. 15, Figure 3 · p. 15, §4.2 · p. 15

4.3 Routing Level Distribution

  • 56.0% of queries routed to Level 2, while 61.5% used Level 1 or 2 and 38.5% required a complete ReAct loop.The reported success rate across all three routing levels was 100%.Source: Table 2 · p. 16, §4.3 · pp. 16–17

4.4.1 Ablation 1: Routing Strategy — Threshold vs. LLM-based Routing

  • The threshold router saved 62.3% of tokens, whereas the LLM router increased consumption by 37.3%, using 272,479 tokens versus the baseline’s 198,508.The LLM router’s 225,628-token routing overhead alone exceeded the baseline’s total token use; threshold routing was zero-cost and deterministic.Source: Table 3 · p. 18, §4.4.1 · pp. 18–19
  • Threshold and LLM-based routing are complementary options, with the choice depending on routing overhead relative to per-query execution cost.The paper argues TRIAGE should support switching between them per scenario, since LLM routing can compensate for threshold routing’s limited generalization when semantic similarity is a weaker reuse signal.Source: §4.4.1 · p. 19

4.4.2 Ablation 2: Component Ablation

  • Without Level 2, token savings fall to 55.0% because parameter-varying queries incur an average 198-token Level 3 cost.This identifies Skill-based parameter substitution as TRIAGE’s primary efficiency driver for structurally similar, parametrically different queries.Source: §4.4.2 · pp. 19–20

4.5 Cross-domain Validation

  • 76.3% token savings across 15 diverse domains (424,792 →100,875 tokens) demonstrate generalization beyond SQL.The L1+L2 reuse rate is 96.2%, with a 3.8% L3 rate; lower savings in more diverse domains include 56.2% for chart lyrics and 65.4% for the cocktail db.Source: §4.5 · p. 20, Table 5 · p. 22
  • In the cross-domain experiment, Level 2 queries used LLM-based reference generation at partial token cost, not the 0-token Skill substitution used in the main SQL experiment, producing an 11.6% L2 ratio versus 56.0%.Source: §4.5 · p. 20

4.6 Online Learning Evolution

  • In ReAct (L3), the L2 hit rate climbs rapidly from 0% to 57% within the first 100 queries, while token cost drops from 198 to 74.7 tokens per query.During warm-up (queries 50–300), the L1 hit rate peaks at 87%, the L3 fraction stabilizes around 4%, and average token cost is approximately 49 tokens per query.Source: §4.6 · pp. 22–23
  • 62.3% token savings over the ReAct baseline are achieved by TRIAGE’s final configuration.The authors describe this as a realistic, measurement-based estimate; the baseline is 197 tokens per query, reflecting real API usage.Source: §4.6 · p. 23

5 Discussion

The discussion identifies structured, repetitive, high-volume settings with predictable tool use as TRIAGE’s best fit, then outlines limitations and research directions for Skill reuse.

5.1 Best Application Scenarios

  • TRIAGE is best suited to structured, repetitive, high-volume queries with predictable execution patterns and limited tool sets.Source: §5.1 · p. 23

5.2 Limitations

  • Reuse is constrained by a 10–50-query cold start, domain-specific trajectories, a Level-3 first query, and possible unverified error propagation.Source: §5.2 · pp. 23–24

5.3 Future Work

  • Program synthesis could replace typed-placeholder induction with executable DSL programs inferred from input-output examples across trajectories.Source: §5.3 · p. 24
  • Future work proposes chaining Skills as planner-selected primitives and unifying SQL, REST, and GUI tools under a common representation.Source: §5.3 · p. 24
  • Lifelong Skill learning must manage redundancy, conflict, and drift through incremental library expansion, deduplication, versioning, and consolidation.Source: §5.3 · p. 24
  • Formal verification, contracts, or symbolic checks at registration are proposed to guard against erroneous Skill patterns and support safe zero-token execution.Source: §5.3 · p. 25
  • Contrastive experience extraction compares failed and corrected SQL, then caches inferred natural-language-to-date-range conversions as Skill formatting rules.Source: §5.3 · p. 25
  • Domain-agnostic trajectory abstraction with quality assessment and pruning is proposed to enable experience sharing across domains and agents.Source: §5.3 · p. 25

6 Conclusion

TRIAGE reuses historical execution trajectories as skills to reduce token use, with substantial savings reported on security-monitoring and cross-domain queries. Its routing-strategy preference depends on the ratio of routing overhead to execution cost.

6 Conclusion

  • 62.3% token reduction (199,782 →75,238) on 1,007 security monitoring queries.The threshold router achieves the same 62.3% savings on the full query set.Source: §6 · p. 25
  • TaaS abstracts experience trajectories into reusable skills.Source: §6 · p. 25
  • 61.5% L1+L2 reuse rate, with 97.6% of queries benefiting from historical trajectory reuse.Source: §6 · p. 26
  • 76.3% token reduction on ToolBench across 15 domains and 345 queries.The paper reports this cross-domain result with real threshold routing.Source: §6 · p. 26
  • Experience-driven execution is presented as a practical way to reduce LLM-agent costs without sacrificing capability.Source: §6 · p. 26
  • The preference for threshold over LLM-based routing depends on the ratio of routing overhead to execution cost.The conclusion motivates a hybrid architecture that adaptively selects the routing strategy.Source: §6 · p. 26

Related papers

Loading 2609.01428v1…