PaperlayerView exact revision on arXiv

Source-linked AI summary

CacheRouter: A Dual-Path Tool Routing Architecture with Cache-Preserving Main-Model Isolation for Long-Tail Tool Discovery

Donghui Zha, Lingwei Xu, Linxiao Wu, Yixue Dong, Haochen Li

arXiv:2608.22708v1cs.AI

TL;DR

Dynamic tool discovery can enlarge prompts and invalidate cached prefixes, creating a conflict between tool flexibility and cache efficiency. CacheRouter separates stable main-model reasoning from independent long-tail tool routing, achieving 95.2% cache hits and reducing input cost to about 8.0% of a no-cache baseline in a 30-turn dialogue.

  • Problem

    Growing or dynamically changing tool definitions increase prompt overhead and can invalidate token-level cache prefixes, limiting simultaneous tool-selection flexibility and cache efficiency.

  • Method

    CacheRouter fixes the main model’s core tool view while an independent routing channel discovers and executes long-tail tools without modifying the main request prefix.

  • Results

    95.2% token-level cache hit rate reduced input cost to about 8.0% of the no-cache baseline in a 30-turn dialogue.

  • Takeaways & Limitations

    The prototype shows that fixed main-model tool definitions can preserve cache-prefix reuse while long-tail tools remain dynamically discoverable and executable.

  • Takeaways & Limitations

    The routing layer is deliberately lossy, discarding tool information through context compression, candidate truncation, and priority filtering to reduce overhead.

Abstract

from arXiv · show

Tool use in LLM systems faces a structural trade-off. Progressive disclosure keeps the prompt small by showing only the tools relevant to the current task, while prompt caching rewards a request prefix that stays fixed across calls; every change to the visible tool list invalidates the cached prefix. This paper treats the trade-off as a problem of request architecture and proposes a dual-path routing design that assigns tool selection and tool delivery to separate channels. The main model always sees a small, fixed set of core tools, so the head of its request is unchanged across calls; all other tools are reached through an independent routing channel, in which a router sub-model searches the full tool list, selects one tool, executes it, and returns the result. Tool registration is automated from source code and supports runtime updates, so the tool set can grow without modifying the main model's request prefix. The design generalizes progressive disclosure: capabilities are disclosed through the routing channel, and the main model's prefix stays stable. A prototype implementation was exercised on 55 functional queries and a 30-turn dialogue; token-level cache hit rates reached 90.99% and 95.2%, cutting input cost to about 12.0% and 8.0% of a no-cache baseline under DeepSeek's pricing, where cache-hit input tokens cost roughly 1/30 of cache-miss tokens.

1 Introduction

CacheRouter addresses the conflict between progressive disclosure and prompt caching by separating tool selection from tool delivery. It keeps the main model’s core-tool view fixed while routing dynamic long-tail capabilities through an independent channel.

  • Motivation: The architecture addresses the opposing pressures of progressive disclosure, which reduces context, and prompt caching, which requires a stable request prefix.Changing tool definitions can forfeit cache-hit discounts even for nearly identical multi-turn tasks.
  • Tool Lifecycle: CacheRouter combines automated registration, hot reloading, and runtime skill bootstrapping so capabilities can grow without synchronized growth of the main model’s cache context.New skills enter an independent dynamic tool space, while the paper explicitly proposes no new retrieval algorithm.
  • Architecture: CacheRouter separates tool selection from tool delivery, preserving a stable main-model request prefix while supporting evolving tool coverage.Traditional orchestration changes the tool-definition segment whenever selection changes, whereas CacheRouter keeps frequent core tools fixed and routes long-tail tools separately.
  • Placeholder Tool Triage: A fixed placeholder tool stabilizes first-round intent triage while signaling that external capabilities exist without changing the main model’s tool-definition prefix.The placeholder’s invocation is explicitly forbidden, and its definition remains unchanged during system operation.
  • INTERNAL ROUTER: INTERNAL ROUTER provides an independent routing channel for tasks beyond the core tool set, preventing dynamic tool discovery from expanding the main model’s fixed view.The routing subsystem is exposed as a pseudo-tool in the main model’s fixed tool view.

2 Related Work

Prior work improves tool-calling capability, retrieval, progressive disclosure, and agent orchestration, but rarely treats changing tool definitions as a prompt-cache problem. CacheRouter instead separates tool selection from main-model tool delivery, keeping core definitions fixed while routing long-tail tools on demand.

  • Tool-use foundations: Early tool-use research established models’ ability to decide when to invoke tools, generate parameters, and use returned results, while later systems improved API selection and invocation reliability.ReAct alternates reasoning and actions, Toolformer learns API use self-supervised, and Gorilla enhances parameter generation and API selection through documentation retrieval.
  • Agent frameworks: Existing frameworks support direct model-to-tool calls and orchestration, but growing tool sets inject changing tool lists into requests and perturb prefix-based caching.LangChain, AutoGPT, HuggingGPT, and LLMCompiler provide engineering abstractions for tool use, planning, and parallel calls.
  • Tool retrieval and progressive disclosure: Large-scale tool systems typically retrieve, rank, or progressively disclose a small candidate set instead of exposing every tool directly.This reduces context burden and decision complexity as tool collections reach hundreds or thousands of entries.
  • Cache-related limitation: Existing retrieval and progressive-disclosure methods optimize selection quality, but changing the selected tool set can still alter request prefixes and prevent continuous cache reuse.Reducing the number of visible tools alone does not guarantee efficiency under cache-based billing.
  • CacheRouter’s distinction: CacheRouter keeps a fixed core tool set in the main model’s request while retrieving long-tail tools through an isolated routing channel, combining stable prefixes with small visible context.The architecture adds cache stability as an orchestration constraint, separates retrieval from request construction, requires no new provider caching mechanism, and can sit atop existing frameworks.

3 System Architecture Overview · 3.1 Design Principles

The architecture separates reasoning from tool delivery across five layers, enabling automatic tool registration, stable request prefixes, and capability evolution without breaking established cache patterns. Its design principles isolate frequently used tools, selection, execution, and runtime updates to preserve cache stability while supporting extensibility.

  • 3.1 Design Principles: Five layers organize reasoning and tool delivery: application, decision, and routing handle the reasoning path, while gateway and repository manage delivery and evolution.The architecture is designed to keep high-frequency request contexts stable as the tool count grows.
  • 3.1 Design Principles: Automatic registration makes newly added directory tools immediately usable, while the deployed agent can write, validate, and consolidate capabilities without breaking existing cache patterns.These requirements operate alongside stable request prefixes as the tool count M increases.
  • 3.1 Design Principles: Information locality separates high-frequency core tools from rarely used long-tail tools into different request contexts, keeping common requests small and stable.The separation is based on usage frequency and limits the tools exposed in high-frequency contexts.
  • 3.1 Design Principles: Forward compatibility requires adding tools without changing existing request contexts or destroying cached prefixes, preserving the value of historical cache results.This principle directly targets cache preservation as the tool ecosystem expands.
  • 3.1 Design Principles: Separating tool selection from execution prevents selection-side changes from perturbing the execution path or its cache.The principle assigns distinct responsibilities to choosing a tool and running it.
  • 3.1 Design Principles: Runtime hot reloading lets tools be added, removed, or updated without restarting the core service because the gateway re-scans the repository and atomically replaces the registry.The implementation and experiments validate reloading itself, while consistency guarantees for in-flight requests during concurrent updates remain future work.
  • 3.1 Design Principles: Together, the principles create a dynamic tool ecosystem that is extensible in function while stable in context and cache behavior on high-frequency paths.The system design names information locality, forward compatibility, separation of concerns, and runtime hot reloading as its four basic principles.

3.2 Overall Framework

The framework uses five isolated layers and staged request processing to keep tool-set changes out of the core request path. An independent router discovers and executes long-tail tools while the main model receives only stable interfaces and structured results.

  • Overall architecture: Five layers isolate application, decision, routing, gateway, and repository complexity behind stable interfaces, preventing tool-set changes from disturbing the core request path.Inter-layer communication uses MCP or inter-process parameter passing, with no shared internal state.
  • Gateway layer: The gateway automatically registers heterogeneous scripts, generates schemas with AST parsing, indexes metadata, and hot-reloads added, removed, or updated tools without restarts.Tool code does not run during registration; runtime reflection is used only as a fallback, while naming and description limits keep metadata compact.
  • Decision layer: The decision layer separates first-round triage, tool execution, and deep-thinking fallback, keeping the full tool set and full history out of any single request.Triage uses a fixed placeholder as a cache anchor; deep thinking responds to status, time, or call-fragmentation anomalies and fallback marks uncertainty rather than fabricating content.
  • Routing layer: The routing layer exposes INTERNAL ROUTER as a pseudo-tool, decoupling tool selection from delivery so the main model sees structured results rather than long-tail definitions.A compressed task summary and the full gateway tool list feed blacklist, whitelist, Top-K, and yellowlist filtering before one constrained tool call.
  • Routing layer: The router is stateless, lightweight, and deliberately lossy, trading context compression, candidate truncation, and priority filtering for lower overhead and a more stable main request path.This design preserves full-tool discoverability while balancing cache stability, tool extensibility, and selection efficiency.

3.3 Core Data Flow

The system triages each query into a direct-answer path or a tool path, while keeping the main model’s visible tool prefix stable. Tool requests proceed through routing, execution, recovery, and persistence in a closed loop.

  • 3.3 Core Data Flow: Triage splits each query into a direct path that answers immediately or a tool path involving gateway startup, execution, recovery, and persistence.The split occurs during first-round triage.
  • 3.3 Core Data Flow: The main model performs triage with only a placeholder tool, enabling direct answers through one API call while preserving a stable tool prefix.No real tool is called during triage.
  • 3.3 Core Data Flow: For tool requests, the gateway registers tools while the main model sees only core packages and INTERNAL ROUTER; routing performs pre-selection without changing that view.Filtering remains inside the routing subsystem.
  • 3.3 Core Data Flow: The execution loop runs ordinary calls concurrently, sends INTERNAL ROUTER calls to the routing sub-model, and stops when a direct answer or completion state is reached.The loop also ends when the round limit is reached.
  • 3.3 Core Data Flow: Nonconvergent loops trigger deep-thinking replanning, while persistent failure produces an uncertainty-marked fallback; completed sessions persist for recovery and tools hot-reload.The overall loop assigns triage, planning, discovery, execution, recovery, and persistence distinct roles.

3.4 Stability Guarantee: Cache Invariance

Cache invariance is achieved by keeping the main model’s core tool set and system prompt fixed while tool-ecosystem changes remain outside its visible request boundary. Append-only history preserves the established prefix, and the routing channel provides dynamic tools without invalidating that prefix.

  • 3.4 Stability Guarantee: Cache Invariance: The design preserves cacheable prefixes by fixing the core tool set C and system prompt while append-only history adds messages only at the end.Cache hits therefore depend on consistency of the existing history prefix, not identical complete payloads.
  • 3.4 Stability Guarantee: Cache Invariance: Runtime tool additions, generated skills, and gateway restarts leave C unchanged because dynamic state remains outside the main model’s visible tool boundary.The registry and router may change freely, but changes reach the main model only as routing results rather than tool definitions.
  • 3.4 Stability Guarantee: Cache Invariance: The architecture separates an extensible dynamic tool space from a stable reasoning space, preventing tool-count growth from translating linearly into main-model request size.INTERNAL ROUTER connects the spaces and exposes long-tail tools on demand.
  • 3.4 Stability Guarantee: Cache Invariance: Static core loading costs O(|C| · savg) rather than O(M · savg), with |C| ≪M, and identical tool segments amortize cache-establishment cost across calls.Tools outside C are not directly callable; the routing channel restores that coverage on demand.

4 Cache-Aware Progressive Disclosure

CacheRouter resolves the conflict between expanding tool coverage and cache stability by separating a fixed core tool view from dynamic long-tail discovery. The main model retains a stable request prefix, while an independent router retrieves, filters, executes, and returns additional tools on demand.

  • CacheRouter partitions tools into a fixed core set visible to the main model and a dynamic long-tail set accessed through an independent routing channel.The core set contains high-frequency, fundamental, and relatively stable tools, while long-tail tools remain available without expanding the main model’s request context.
  • The main model invokes INTERNAL ROUTER when its core tools are insufficient, after which a sub-model retrieves, filters, executes, and returns one appropriate long-tail tool result.INTERNAL ROUTER has no business logic and serves as the entry point to the dynamic tool space.
  • From the main model’s perspective, the visible tools remain C ∪{INTERNAL ROUTER} regardless of the total registry size M, preserving access to newly added tools without changing its tool definitions.Dynamic tool information propagates only inside the routing subsystem, implementing progressive disclosure through the router.
  • Router-side refresh, filtering, and execution never modify the main model’s tool definitions, so append-only history preserves the existing request prefix for cache reuse.This is an architectural guarantee about request structure rather than a promise about the actual answer.

5 Experimental Evaluation · 5.1 Experimental Setup and Evaluation Metrics

The experimental evaluation validates a CacheRouter prototype across eight functional experiments and a 30-turn dialogue, using 55 query cases plus 30 dialogue cases. It measures task behavior, routing activity, latency, and token-level cache performance across all API calls.

  • 5 Experimental Evaluation: The prototype validation covers eight main execution paths and a dedicated 30-turn dialogue experiment measuring cache hits and multi-turn stability.The functional paths include sequential, parallel, long-chain, triage, convergence, fallback, router filtering, and skill-bootstrapping tasks.
  • 5.1 Experimental Setup and Evaluation Metrics: Experiments 1–8 contain 55 query cases, while the long-chain dialogue contains 30 query cases executed sequentially in one session.The system runs in Linux using Python and the MCP Gateway Service; both models call the DeepSeek API.
  • 5.1 Experimental Setup and Evaluation Metrics: Task completion rate measures the proportion of queries that converge successfully to a final answer.This metric captures whether a query completes successfully rather than how many execution steps it requires.
  • 5.1 Experimental Setup and Evaluation Metrics: Execution rounds count model reasoning rounds in the tool loop, with direct-answer tasks assigned 0 rounds.The metric distinguishes direct responses from queries requiring iterative tool execution.
  • 5.1 Experimental Setup and Evaluation Metrics: Router invocations count INTERNAL ROUTER calls by the main model, reflecting use of the long-tail routing channel.Per-query latency separately measures wall-clock time from query entry until final-answer generation.
  • 5.1 Experimental Setup and Evaluation Metrics: Token-level cache hit rate is the core metric and is computed at token rather than query granularity.Cache data cover all API calls, including every tool-loop round and routing-submodel request; DeepSeek cache-hit input tokens cost about 1/30 of cache-miss tokens.
  • 5.1 Experimental Setup and Evaluation Metrics: Token-level cache hit rate equals cached input tokens divided by cached plus uncached input tokens.The cost calculation uses official DeepSeek prompt-cache hit and miss token fields and a cache-hit/cache-miss billing ratio of about 1:30.

5.2 Experiments 1–8: Functional and Stability Evaluation

Experiments 1–8 evaluated functional coverage, stability, fallback, filtering, and skill bootstrapping across 55 queries, all of which succeeded. Aggregate cache hit rate reached 90.99%, reducing cached input cost to about 12.0% of the no-cache baseline while preserving the main model’s request-prefix stability.

  • Experiment 1: Sequential tasks: All five sequential tasks completed successfully, with the router locating the /tmp sandbox and replanning when deployment whitelists blocked requested file paths.This demonstrates adaptation to execution-environment constraints during multi-step tool chains.
  • Cache stability and skill bootstrapping: The fixed main-model tool view remained unchanged across gateway restarts, while direct-answer paths benefited most from caching and multi-round debugging chains benefited least.Experiment 3’s cold-start comparison confirms prefix stability, whereas Experiment 8’s changing prefixes limit cache benefits.
  • Experiment 7: Router triggering and filtering: All 15 filtering-configuration queries succeeded, with only 4 Router invocations, while arithmetic tasks were answered directly and the long-tail script completed in every configuration.The result supports the core set as the default path and the Router as a long-tail supplement; a high-latency case converged after retry.
  • Overall functional evaluation: 55/55 queries succeeded, including long-tail tool execution, honest unanswerable-query fallback, skill bootstrapping, and complex or anomalous cases requiring replanning.The Router was invoked 40 times, while unanswerable scenarios and skill bootstrapping each succeeded in 5/5 cases without fabrication.
  • Cache stability: 90.99% of input tokens hit the server-side cache, reducing cached input cost to about 12.0% of the no-cache baseline under DeepSeek’s approximately 1:30 hit-to-miss pricing.Table 9 aggregates token-level cache statistics across all eight experiments; the result reflects structural prefix stability rather than a guaranteed server-side hit rate.

5.3 Long-Chain Multi-Turn Dialogue Experiment

In a 30-turn single-session dialogue, CacheRouter maintained a 95.2% token-level cache hit rate and reduced input cost to about 8.0% of the no-cache baseline. The experiment further showed stable prefix reuse, prompt-context reclamation, and substantial miss-side cost that could be reduced by direct-answer triage.

  • 5.3 Long-Chain Multi-Turn Dialogue Experiment: 95.2% cache hit rate and 8.0% no-cache input cost were achieved across 30 turns, with all cases converging successfully and only one Router invocation.The result improved over the 12.0% aggregate input-cost ratio for Experiments 1–8.
  • 5.3 Long-Chain Multi-Turn Dialogue Experiment: 95.2% cache reuse persisted under append-only history growth, with misses confined to newly appended tails while fixed prefixes remained cached.The 30 queries ran sequentially with continuously accumulating history, confirming stable long-chain prefix caching.
  • 5.3 Long-Chain Multi-Turn Dialogue Experiment: Cache hit proportions increased across execution rounds because later rounds mostly reused established prefixes while adding only tail tokens.The 14 miss queries each entered 2–3 tool-execution rounds, causing hit tokens to accumulate over rounds.
  • 5.3 Long-Chain Multi-Turn Dialogue Experiment: 11.3K-token execution peaks were reclaimed before the next query, returning the prompt to 2,483 tokens and preventing cross-query cache-prefix pollution.The peak occurred at Q29 round 3, while Q30’s first round returned to the question–answer-history scale.
  • 5.3 Long-Chain Multi-Turn Dialogue Experiment: 24× higher input consumption occurred for the cache-miss case than the comparable cache-hit case, and miss queries contributed about 87% of input tokens.Q1 consumed 1,002 tokens, whereas Q25 consumed 23,875 tokens; direct-answer triage is proposed to reduce such miss-side costs.

6 Conclusion and Future Work

CacheRouter resolves the conflict between dynamic tool sets and prompt caching by isolating long-tail tool retrieval and execution in an independent routing channel while keeping the main model’s core tool view fixed. The prototype demonstrates this cache-stable architecture, while future work targets scalability, retrieval quality, and routing efficiency.

  • Conclusion: CacheRouter keeps the main model’s core tool view fixed while an independent routing channel retrieves and executes long-tail tools, preserving cache stability as the ecosystem evolves.The architecture uses a placeholder tool, AST-based registration, hot reloading, and skill bootstrapping without modifying the main cache path.
  • Limitations: The approach remains limited by keyword pre-selection, routing-model reasoning cost and latency, cache misses on closed-book questions, and unexplored server-side cache strategies.These limitations define the paper’s future-work agenda.
  • Future Work: Multi-layer progressive disclosure could scale routing beyond tens of thousands of tools by recursively narrowing candidates before the final router.The proposed hierarchy extends the current dual-layer core-tool-plus-single-router design to reduce candidate-compression pressure.
  • Future Work: Vector retrieval could improve pre-selection recall for semantically related tools and reduce the candidate set processed by the routing sub-model.The proposal encodes tool names, descriptions, and parameter schemas, then retrieves Top-K candidates using the task context.
  • Future Work: Multi-step routing-sub-model planning could execute tool-selection loops within the independent channel, reducing repeated main–sub-model round trips on complex tasks.The current router performs one constrained selection and execution per call, requiring repeated INTERNAL ROUTER invocations for complex tasks.
  • Conclusion: The prototype demonstrates a unified architecture for tool registration, routing, skill bootstrapping, and cache isolation under the constraint of cache stability.The reported system is explicitly described as a working prototype.
Loading 2608.22708v1…