PaperlayerView exact revision on arXiv

Source-linked AI summary

Nexus: Depth-Adaptive KV-Cache Splicing and Retrieval-Decoupled Tool Routing for Agentic LLMs on Unified Memory

Mustafa Arslan

arXiv:2608.20397v1cs.AI

TL;DR

Nexus addresses the growing prefill cost of verbose MCP tool schemas by decoupling routing from schema KV-cache processing and repairing only the depth-dependent splice residue. Retrieval-based routing over compact textual signatures provides the main path, while depth-adaptive suffix recomputation bounds the secondary splice path. The measured system reaches a 1.66× faster first argument with approximately 80% main-context token savings and preserves output fidelity, but its quantitative envelope is limited to a specific model and hardware setup.

  • Problem

    Verbose MCP tool schemas are re-encoded every turn, making quadratic attention-prefill cost dominate latency as tool registries grow.

  • Method

    Nexus decouples tool routing through an INT8 semantic lookaside buffer and compressed textual signatures, while using depth-adaptive suffix recomputation to repair relocated schema KV blocks.

  • Results

    1.66× first-argument latency reduction and approximately 80% main-context token saving are achieved versus full-schema re-prefill, while repaired splices preserve top-1 agreement and DKL ≈ 0.

  • Takeaways & Limitations

    Retrieval-decoupled routing is the more durable path, while KV splicing offers moderate-depth TTFT gains that converge to prefill parity at deep context.

  • Takeaways & Limitations

    The quantitative envelope is tuple-specific, with small samples, UMA-only physical splicing, and deep-splice validation limited to fidelity and latency rather than multi-turn production use.

Abstract

from arXiv · show

Agentic large language models (LLMs) on the Model Context Protocol (MCP) re-encode verbose tool schemas every turn, so prefill - quadratic in sequence length - dominates time-to-first-token (TTFT) as the tool registry grows. Nexus's primary lever is to decouple routing from the schema-prefill cost: an INT8 semantic lookaside buffer (SLB) with a calibrated cross-encoder margin gate selects tools by retrieval, and arguments are generated over a compressed textual signature (median 19 tokens) rather than over spliced key/value (KV) cache. This path is depth-independent: routing accuracy stays near 89% as the registry scales to 250 tools - where a concatenate-all-schemas baseline overflows the context window entirely - and it reaches a first-argument token 1.66x sooner than a full-schema re-prefill at a ~80% main-context token saving. As a secondary, bounded lever we transplant a compiled schema KV block directly into the live context. This is fundamentally limited by rotary position embedding (RoPE) phase drift: an anchored splice is output-exact, but off-anchor placement corrupts attention, so beyond a threshold P=256 Nexus repairs the seam with a depth-adaptive suffix redecode that escalates to a full re-prefill. The resulting never-regress property is a guarantee on output fidelity (top-1 agreement, D_KL approx. 0) - not on latency, which can dip to 0.98x before converging to parity - alongside a 1.1-1.7x TTFT speedup at moderate depth that narrows to parity at deep context. Two negative results bound the design: the off-anchor RoPE fidelity boundary, and the failure of a reference-free drift gate to predict drift (Spearman rho = 0.193). All measurements are from one model tuple (Qwen2.5-14B-Instruct Q4_K_M) on Apple-silicon unified memory; the qualitative boundaries generalize, while the quantitative envelope is tuple-specific.

I. INTRODUCTION

Nexus addresses the schema-prefill wall in MCP agents by separating tool routing from schema KV-cache splicing. It combines retrieval-based routing with bounded, depth-adaptive repair while measuring scope on a specific Apple-silicon model setup.

  • Tool schemas dominate growing MCP prompts, making every-turn prefill a poorly scaling cost because self-attention is O(N^2).
  • Nexus routes tools through an INT8 semantic lookaside buffer and calibrated cross-encoder gate, generating arguments from compressed textual signatures.
  • At 250 tools, retrieval-decoupled routing remains usable while the concatenate-all-schemas baseline overflows the context window.
  • The hybrid path reaches the first argument token 1.66× sooner than full-schema re-prefill while saving approximately 80% of main-context tokens.
  • The prototype is evaluated primarily on Qwen2.5-14B-Instruct Q4_K_M with Apple-silicon unified memory, using small end-to-end samples and a non-production deep-splice path.

II. RELATED WORK

Prior approaches retrieve or compress tool schemas and manage KV caches in place, whereas Nexus relocates compiled per-tool blocks and decouples routing from the splice path.

  • Retrieval-augmented selection and schema compilation reduce the schema presented before prefill, while learned latent retrieval requires trained alignment.
  • Paged, prefix-shared, and dynamic virtual-memory KV systems reuse cache in place rather than relocating blocks to new positions.
  • Nexus differs by relocating coarse per-tool KV blocks and treating routing as separate from the splice path.
  • Scaled dot-product attention has O(N^2d) time and O(N^2) attention memory per layer, locating the dominant sequence-length cost.
  • Because schemas are re-encoded every turn, both concatenating all schemas and retrieving K schemas still pay schema-prefill cost online.

B. RoPE Phase Drift

RoPE makes relocated KV blocks position-sensitive: off-anchor placement introduces phase drift, while reanchoring restores relative query-key rotation but leaves residual context-dependent divergence.

  • RoPE phase error grows with placement offset and varies across feature dimensions, so compiled schema blocks are faithful only near their compile anchor.
  • The splicer declines a bare splice beyond max_splice_pos_ = 256, treating the threshold as the start of repair rather than an absolute failure wall.
  • Nexus re-rotates transplanted keys from compile anchor m0 to runtime position npast so subsequent queries read the relocated block on-axis.
  • Nexus relocates per-tool blocks unlike in-place KV-serving systems, so comparisons with non-Nexus numbers are contextual rather than direct performance claims.
  • Reanchoring exactly recovers relative query-key rotation but cannot remove divergence caused by the block’s different preceding context.

C. Depth-Adaptive Recompute

Nexus repairs increasingly deep splices by redecoding a growing trailing suffix rather than treating the 256-token threshold as a hard wall. The recompute fraction eventually reaches full-prefill behavior.

  • Beyond M = 256 tokens, Nexus redecodes a trailing schema suffix whose fraction grows with context depth.
  • The repair begins at Rbase = 5% and reaches 100% at a configurable multiple K of the threshold.
  • At deep positions, recomputation approaches full text prefill with DKL(p0 ∥ psplice) = 0.

IV. PHYSICAL LIMITATIONS AND NEGATIVE RESULTS

Nexus finds that off-anchor KV splicing produces bounded distributional drift rather than a sharp fidelity cliff, while scattered partial recomputation can severely corrupt outputs. These negative results motivate contiguous-suffix repair beginning at P=256.

  • The design presents these negative results first because they determine the repair curve and require routing to remain depth-independent.
  • Anchored splicing is output-exact at Δpos = 0, but off-anchor RoPE phase error perturbs the next-token distribution.Reanchoring reduces the residual to a small band rather than eliminating it.
  • P=256 is a conservative repair-start: contiguous-suffix bare splicing remains in the 0.008–0.038-nat D_KL band with top-1 agreement = 1.0 through Δpos = 2048.The threshold marks separation from the anchor floor, not a fidelity cliff.
  • DKL ≈5.7 nats at npast = 1024 made retired LegoLink scattered recomputation two orders of magnitude worse than contiguous-suffix bare splicing.The result supports repairing a contiguous trailing suffix instead of scattered regions.

B. The Failure of Reference-Free Drift Gating

Nexus tests a cheap reference-free K-variance proxy for deciding when deep splices drift, but finds that it cannot reliably rank the observed per-head drift.

  • ρ = 0.193: per-head preceding-context K-variance fails to rank-correlate with true drift, below the 0.40 target.Observed drift still varies, with maxima of 175–207 and means of 91–105 across depths and contexts.
  • A scalar-threshold gate based on K-variance would fire indiscriminately because the proxy cannot separate high- from low-drift heads.
  • Nexus therefore retains a deterministic depth-adaptive repair curve instead of using the reference-free proxy.

B. Execution Sidecar and Compressed IR

Nexus separates routing and argument generation from deep-context splice risk through a sidecar sequence and compressed textual tool signatures. Compiled KV blocks remain a separate, depth-bounded acceleration path.

  • Execution Sidecar and Compressed IR: A temporary sidecar beginning at position 0 keeps candidate splices in the output-exact regime and preserves coreference with a pruned sliding window.Recent turns remain verbatim while bulky tool payloads become short semantic surrogates.
  • Execution Sidecar and Compressed IR: Compiled tool blocks store RoPE metadata followed by page-aligned contiguous F16 keys and values, which are written at the runtime cursor before suffix redecode.
  • Execution Sidecar and Compressed IR: Routing depends only on the query embedding and tool registry, while arguments use a compressed textual signature rather than spliced KV.
  • Execution Sidecar and Compressed IR: Soft-capped architectures require layout-aware transposed-V copying, validated against live strides before physical-cache writes.Nexus fidelity-tests this path on Gemma-2-9B.

C. Cache-Line Hardening and Concurrency

The implementation hardens concurrent execution with coarse-grained turn serialization and cache-line isolation, while reusing warm prefixes through a fixed zero-allocation sequence cache.

  • Cache-Line Hardening and Concurrency: A single coarse re-entrant lock serializes each turn, producing deterministic, bit-stable outputs under concurrency.Fine-grained Python locking would race the native C++ decode mutex.
  • Cache-Line Hardening and Concurrency: alignas(128) and a mutex pinned to its own cache line reduce allocator false-sharing, but Nexus reports no isolated throughput speedup.
  • Cache-Line Hardening and Concurrency: A fixed pool of 32 sequence slots reuses warm prefixes by exact-token longest-common-prefix matching without hot-path allocation.

VII. EVALUATION

Nexus evaluates depth-adaptive KV splicing and retrieval-decoupled routing on Apple-silicon unified memory, measuring fidelity, TTFT, routing accuracy, and registry-scale behavior. Retrieval remains effective as tool count grows, while splice speedups narrow toward parity as recomputation increases.

  • Depth-Adaptive Splice: 1.1–1.7× TTFT speedup appears at moderate depth, but the deep splice converges to approximately 1.0× parity as recomputation approaches 100%.The default K = 4 and tuned K = 16 curves trade deeper sustained speedup against larger repaired suffixes.
  • Depth-Adaptive Splice: Top-1 next-token agreement and D_KL(p0 ∥ psplice) ≈0 hold across deep-splice cells, while latency can dip to 0.98× before recovering to parity.The never-regress guarantee concerns output fidelity rather than latency, with 15 trials per cell.
  • Routing Accuracy and Registry Scale: 92%, 90%, 89%, and 89% routing accuracy are reported at N = 10, 50, 100, and 250 tools, respectively.At N = 250, the Wilson 95% CI is [81.4, 93.7] over 100 queries.
  • Retrieval-Decoupled Routing: 1.66× first-argument latency reduction accompanies approximately 80% main-context token saving for retrieval-decoupled routing.The evaluation uses Qwen2.5-14B-Instruct Q4_K_M on an Apple M4 Max system with 64 GB unified memory.
  • Routing Accuracy and Registry Scale: At N = 250, the concatenate-all-schemas oracle overflows the context window, whereas Nexus continues routing with a median 19-token routing payload.The oracle reaches 98% at N = 10, but its result is undefined once the prompt no longer fits.

C. Argument Fidelity

The argument-fidelity evaluation tests compressed textual routing and argument generation at small sample sizes. It reports routing, argument, JSON-validity, and placeholder-leakage outcomes, while noting that coreference accuracy was not separately measured.

  • Argument Fidelity: 86.7% sidecar routing accuracy is reported on a 30-case consistency set using a median-19-token compressed textual IR.The Wilson 95% CI is [70.3, 94.7], with p99 IR length of 32 tokens.
  • Argument Fidelity: 100% of query-specified arguments were filled correctly on routed cases, with 100% JSON validity and no placeholder leakage.The argument result covers 40 specified arguments, with a 95% CI lower bound of ≥91.2%.
  • Argument Fidelity: 80% end-to-end argument accuracy is reported when routing errors are counted as failures.This corresponds to 40/50 arguments, with Wilson 95% CI [67.0, 88.8].
  • Evaluation Boundary: No separate coreference accuracy is reported because that measurement is absent from the committed v2.0 bundle.The table materials identify routing and micro-cost measurements at N = 250 tools, but do not add a coreference result.

D. Discussion

Nexus supports a bounded acceleration thesis: retrieval-decoupled routing provides the durable path, while depth-adaptive splice repair offers moderate-depth gains that decay to parity. The claims are quantitatively constrained to a single measured model tuple and small-sample evaluation, although the qualitative RoPE boundaries are presented as broader.

  • Measured envelope: 1.1–1.7× TTFT reduction occurs at moderate depth, decaying to parity at deep context.The evaluation describes this as a bounded result rather than an unconditional speedup.
  • Limitations and scope: All quantitative numbers come from one Apple M4 Max UMA host and one Qwen2.5-14B-Instruct Q4_K_M tuple, so generality beyond these settings is unproven.The transposed-V mechanism receives an additional Gemma-2-9B fidelity check, but the quantitative envelope remains tuple-specific.
  • Measured envelope: Near 89% routing accuracy persists through 250 tools, while the concatenate-all baseline cannot route at N ≥50 after overflowing the context window.This supports retrieval-decoupled routing as the more scalable path.
  • Limitations and scope: The never-regress curve is measured at fidelity-and-latency level, not exercised as a multi-turn production path.End-to-end arms use n ≤30, and deep splice is microvalidated.
  • Generalization: Qualitative boundaries are presented as model-agnostic RoPE properties, whereas the P = 256 threshold, recompute constants, and divergence magnitudes require re-measurement.The stated boundaries include off-anchor drift, limited scattered-recompute repair, and reference-free gating failure.
Loading 2608.20397v1…