PaperlayerView exact revision on arXiv

Source-linked AI summary

UNISON: A Co-Designed Near-Memory Scheduler of Session KV Residency for LLM Agents

Fan He, Yan Li, Xiaoyang Zeng

arXiv:2609.09643v1cs.AR

TL;DR

Agent loops retain growing KV prefixes across tool waits while many sessions compete for finite SRAM/HBM, making eviction and tier placement a session-level efficiency problem. UNISON shares one live ranking between eviction and idle-window migration; across six traces it is the best non-oracle policy on hit rate and AMAT, with lower TTFT on long-horizon traces.

  • Problem

    Agent sessions retain growing KV prefixes across tool waits while sharing finite SRAM/HBM, and existing recency-, timeout-, or identity-based proxies miss loop mechanism information.

    Source: §VII · p. 12

  • Method

    UNISON is an event-driven near-memory scheduler that shares a ranking between SPEAR eviction scoring from gap history and turn-indexed hazard, and TIDE migration using observed waits as DMA budgets.

    Source: §I · p. 2

  • Results

    Across six traces, UNISON is the best non-oracle policy on hit rate and AMAT, raising hit rate by 0.3% to 23.1% and reducing AMAT by 22% to 51%, while lowering long-horizon TTFT by 58% to 89%.

    Source: §I · p. 2, §VII · p. 13

  • Takeaways & Limitations

    The authors’ structural analysis finds that the unified near-memory control plane cannot be decomposed into independent IPs or realized in software without reintroducing documented failure modes.

    Source: §I · p. 2

    Retained prefixes can improve hit rate and AMAT yet leave residual prefill on the critical path, allowing AGSERVE to achieve marginally lower aggregate TTFT/LRU ratio.

    Source: B. Main Results · p. 8

Abstract

from arXiv · show

Large language models are increasingly composed into agent loops that plan, call tools, and resume the same task after each action. These loops press a shared memory hierarchy harder than conventional multi-turn chat, because they hold a growing key-value (KV) prefix across tool waits and place many sessions on one SRAM/HBM pool, so that eviction and hierarchical placement become a session-level efficiency problem orthogonal to compute-mode optimization. Existing proxies based on recency, timeout, or identity miss the mechanism information of the loop and therefore treat a live wait as a cold, discardable unit. We present Unified Native Inter-turn Session Orchestration Nexus (UNISON), an event-driven near-memory scheduler in which Survival-Penalty Eviction for Agent Return-gap (SPEAR) and Tiering in Idle-window DMA Events (TIDE) share one live ranking. SPEAR selects who leaves from a gap average and a turn-indexed hazard, while TIDE spends the observed wait as a DMA budget for who sits in the fast tier. On coding and general-mission benchmarks with three model families, totaling 1,415 sessions and 33,596 turns, the joint policy is the best non-oracle entry on every trace, raising hit rate by 0.3% to 23.1%, reducing AMAT by 22% to 51%, and lowering TTFT by 58% to 89% on long-horizon traces. A structural necessity analysis shows that the unified near-memory design cannot be decomposed into independent IPs or realized in software without re-introducing documented failure modes. The 28-nm CMOS scheduling core occupies 0.169 mm^2 at 13.6 mW and 150 MHz, a negligible overhead relative to the KV hierarchy it manages, reproducing the floating-point ranking at Kendall tau exceeding 0.998.

I. INTRODUCTION

Agent loops retain growing KV prefixes across scheduled tool waits while many sessions compete for finite SRAM/HBM, making session-level eviction and tier placement an efficiency problem. UNISON uses loop signals for both decisions; its joint policy outperforms non-oracle alternatives across six traces, while existing proxies miss information in those signals.

I. INTRODUCTION

  • Agent loops retain growing prefixes across scheduled tool waits, with many sessions sharing a finite SRAM/HBM pool, so residency becomes a session-level efficiency problem.The live set can exceed provisioned memory envelopes across edge and cloud workloads.Source: §I · pp. 1–2

A. Information-Conditioned Eviction Methods

  • Recency and timeout treat scheduled tool waits as cold, while other proxies can make per-request decisions, homogenize tool waits, or lose ranking power on uniform workflows.These approaches do not consistently use each session’s gap history and progress toward completion.Source: §I · pp. 1–2

I. INTRODUCTION

  • UNISON places an event-driven scheduler beside the hierarchy, using SPEAR to select who leaves and TIDE to select who occupies the fast tier from a shared ranking.Both decisions use loop mechanism information rather than task content.Source: §I · p. 2
  • Across six traces, UNISON is the best non-oracle policy on hit rate and AMAT, raising hit rate by 0.3% to 23.1% and reducing AMAT by 22% to 51%.On long-horizon traces, it also lowers TTFT by 58% to 89% when extra prefill sits on the critical path.Source: §I · p. 2

A. Information-Conditioned Eviction Methods

  • Prior methods illustrate distinct limits: block recency evicts waiting sessions, per-request lifecycle rules do not rank the shared live set, and identity- or workflow-conditioned scores flatten on homogeneous batches.Token-pruning and datapath approaches reshape one request’s working set rather than rank live sessions against one another.Source: A. Information-Conditioned Eviction… · p. 2

B. Hierarchy-Aware Placement Strategies

Existing hierarchy-placement strategies use static plans, scheduler hints, or popularity rather than each session’s current gap. UNISON’s session-level scheduler complements datapath engines, while its workload model frames online eviction and tier placement under causal information and finite capacity.

B. Hierarchy-Aware Placement Strategies

  • Static plans, scheduler hints, and prefix-popularity ordering do not condition placement on the gap a session has just opened, so they can place retained prefixes on the slow path.The cited approaches include FlexGen, CachedAttention, and Mooncake, respectively.Source: B. Hierarchy-Aware Placement Strategies · p. 3

C. Near-Memory and Datapath KV Hardware

  • A near-memory scheduler ranks session residency on request and gap events, complementing datapath engines that prune tokens within the request in flight.The two mechanisms do not substitute for each other.Source: C. Near-Memory and Datapath KV Hardware · p. 3

III. METHODOLOGY

  • The methodology separates the online control problem into two-tier capacity and cost, causal eviction scoring, gap-budgeted tier placement, and a unified event-driven loop.The pipeline is presented in Fig. 2.Source: §III · p. 3

A. Workload Model and Problem Formulation

  • The workload model treats each execution as a multi-turn session whose later turns inherit most of the earlier prefix, making a miss costlier than a hit.The live set is constrained by SRAM/HBM capacities and a session-slot limit; online policies use causal signals and cannot observe future arrivals.Source: A. Workload Model and Problem… · p. 3

B. SPEAR: Survival-Penalty Eviction for Agent Return-Gap

SPEAR estimates next-reference distance from a session’s gap recurrence and turn-indexed completion hazard. Its composite score raises eviction priority for sessions expected to wait longer or nearing completion.

B. SPEAR: Survival-Penalty Eviction for Agent Return-Gap

  • SPEAR combines a gap exponential moving average with a turn-indexed completion hazard to rank residents by estimated next-reference distance.The two signals capture different timescales: expected duration of the current gap and whether the session will return at all.Source: B. SPEAR: Survival-Penalty Eviction… · pp. 3–4
  • A long smoothed gap indicates a session is expected to remain idle longer, so it should be evicted before one that returns frequently.The gap estimate is updated with coefficient α=3/10.Source: B. SPEAR: Survival-Penalty Eviction… · p. 4
  • The score’s expected-remaining-wait term is scaled by survival likelihood, and a completion penalty grows as survival falls, raising eviction priority near completion.The completion penalty P is programmable; the local complement σ(t) needs one table lookup rather than a sequential cumulative estimator.Source: B. SPEAR: Survival-Penalty Eviction… · p. 4
  • SPEAR addresses eviction ranking; deciding where surviving sessions reside in the two-tier hierarchy is a complementary placement problem.Fig. 2 depicts the broader pipeline from survival-based scoring to idle-window migration.Source: B. SPEAR: Survival-Penalty Eviction… · p. 4, Figure 2 · p. 4

C. TIDE: Tiering in Idle-Window DMA Events

TIDE uses observed tool-gap windows as DMA budgets to migrate sessions between SRAM and HBM. It shares SPEAR’s ranking with eviction in an event-driven control loop, so the two decisions use one state and score.

C. TIDE: Tiering in Idle-Window DMA Events

  • TIDE turns an idle tool gap into a DMA budget and migrates sessions so the lowest-scoring residents occupy the fast tier when the next request arrives.The budget is based on estimated remaining gap duration and logical DMA bandwidth, subject to legal capacity-preserving swaps.Source: C. TIDE: Tiering in Idle-Window DMA… · pp. 4–5

D. Joint Online Control Loop

  • TIDE uses the same SPEAR score for migration candidates as eviction, coupling placement quality to eviction quality rather than maintaining independent ranking state.The loop updates a shared register file on each event, scans for eviction on capacity violations, and initiates migration when a gap is open and DMA is free.Source: C. TIDE: Tiering in Idle-Window DMA… · p. 5, D. Joint Online Control Loop · p. 5

IV. ARCHITECTURE

UNISON places a session-level control plane beside the KV hierarchy, where one live ranking supports eviction and tier placement. Its shared state addresses interleaved events, while fixed-point scoring preserves the reference ranking with low drift.

A. Controller Scope and System Placement

  • The controller owns session-residency decisions, while software supplies events and receives a victim index and logical DMA descriptors; KV storage and physical memory interfaces remain external.Near-memory placement observes events at cycle granularity, avoiding the polling or interrupt overhead of a software scheduler.Source: A. Controller Scope and System Placement · p. 5

B. Unified Control Plane

  • A unified register file is structurally necessary because fine-grained agent events make eviction and promotion reads of one ranking, not independent computations.The dual-IP design diverges even with every-event synchronization; reduced synchronization causes hit-rate drops up to 5.4 pp and inconsistent AMAT signs across traces.Source: B. Unified Control Plane · pp. 5–6, Table I · p. 6
  • The unified pipeline sequences SPEAR before TIDE, letting both mechanisms read the same live register-file state.The architecture integrates ranking and migration beside the two-tier KV hierarchy.Source: B. Unified Control Plane · p. 6, Figure 3 · p. 6

C. Fixed-Point Quantization of the Scoring Contract

  • A single ROM lookup of the local complement σ(t) avoids the sequential dependence chain of a cumulative survival estimator in the scoring pipeline.The selected integer format achieves aggregate Kendall τ above 0.998 with worst-cell hit-rate drift of 0.192 pp; 12-bit hazard is the minimum width passing the ranking gate.Source: C. Fixed-Point Quantization of the… · pp. 6–7
  • The format sweep evaluates 180 configurations on 7,265 real eviction snapshots, using top-1 victim agreement ≥0.99 and maximum hit-rate drift ≤0.20 pp as hard gates.The sweep varies hazard and gap widths, time scale, divider type, and penalty form.Source: C. Fixed-Point Quantization of the… · p. 7

D. Event Core Microarchitecture

The event core uses an area-conscious pipelined scan to produce decisions within the tool-gap window. Cosimulation confirms that the locked format matches the floating-point reference on the tested trace.

D. Event Core Microarchitecture

  • A pipelined linear scan completes in N+5 cycles, trading latency for area that grows with the register file instead of dedicated comparator logic.A parallel comparator tree would close in O(1) cycles but scale as O(N) in area.Source: D. Event Core Microarchitecture · p. 7
  • The five-stage pipeline maps to the quantized-score arithmetic, with the critical-path divider determining its depth.The controller must finish each live-session scan within the tool-gap window so a decision is ready before the next request.Source: D. Event Core Microarchitecture · p. 7
  • Cosimulation is bit-exact against the floating-point reference across all 1,937 events and 845 decisions of the GAIA/Gemma4 Edge-Tight trace.Fig. 4 explores session capacity and structural variants using the Yosys/Nangate45 synthesis flow.Source: D. Event Core Microarchitecture · p. 7, Figure 4 · p. 7

E. Session-Capacity Design-Space Exploration

The capacity sweep balances policy effectiveness against decision latency and register-file area. The selected knee is N=64, where effectiveness saturates while latency remains below the median tool gap.

E. Session-Capacity Design-Space Exploration

  • ASE is the six-dataset geometric mean used to evaluate policy effectiveness across session capacities, with the inner product spanning three normalized metrics.The capacity sweep uses this measure to compare policy effectiveness.Source: E. Session-Capacity Design-Space… · p. 7
  • ASE saturates near N=64 as trace concurrency plateaus, while decision latency remains well below the median tool gap.Fig. 4 reports the capacity exploration on the Yosys/Nangate45 synthesis flow.Source: E. Session-Capacity Design-Space… · p. 7
  • At N=64, the five live configurations occupy a narrow cell-count band, and the selected design retains the exact divider for bit-exact scoring.The reciprocal-LUT alternative ties ranking fidelity without improving the end-to-end gate.Source: E. Session-Capacity Design-Space… · p. 7

V. SOFTWARE EVALUATION

Across six agent traces, UNISON is the best non-oracle policy on hit rate and AMAT, while TTFT outcomes depend on the workload’s queueing regime. The evaluation uses a replay corpus spanning two task suites and three model families.

A. Experimental Setup and Metrics

  • The evaluation replays six SWE-bench and GAIA traces across Qwen3-Coder-30B, Devstral-24B, and Gemma4-E4B, totaling 1,415 sessions and 33,596 turns.It reports hit rate and prefill reduction as arithmetic means across six capacity envelopes, and AMAT/LRU and TTFT/LRU as geometric-mean ratios; TTFT is p50 under 4× load.Source: A. Experimental Setup and Metrics · p. 8

B. Main Results

  • UNISON is the best non-oracle entry on hit rate and AMAT on every dataset, with BR=0.93 and a mean AMAT reduction of 34.8% versus LRU.It outperforms CacheScout on hit rate across all six datasets; the remaining 7% gap to Bélády reflects online uncertainty about the next arrival.Source: B. Main Results · p. 8
  • AGServe’s aggregate TTFT/LRU ratio is marginally lower by 0.01 point, although UNISON leads on hit rate and AMAT.The serving model charges leftover prefill to the head-of-queue request; retaining prefixes can leave more residual prefill on a competing arrival’s critical path.Source: B. Main Results · p. 8
  • When long prefills saturate GPU compute on SWE/Qwen3 and SWE/Devstral, queuing delay dominates and policy ordering ceases to influence TTFT.The cache-bound regime where eviction policy most directly improves latency is isolated in the vLLM validation.Source: B. Main Results · p. 8

C. Scoring and Tiering Ablation

The ablations show that scoring and tiering work best together, while hazard and gap recurrence are the most consequential SPEAR signals. UNISON’s advantage persists across tested storage capacities, and the hazard LUT is stable across session splits and converges online.

C. Scoring and Tiering Ablation

D. Signal Contribution Analysis

  • Hazard and gap recurrence are indispensable SPEAR signals; elapsed idle has the smallest mean hit-rate impact at 1.55 pp and is omitted from RTL.Elapsed idle is the only signal requiring a per-slot hardware timer, supporting its omission on accuracy and area grounds.Source: D. Signal Contribution Analysis · p. 8, Figure 6 · p. 9

E. Robustness Across Capacity Operating Points

  • UNISON’s six-trace mean gains in hit rate, AMAT, and prefill reduction remain positive at every tested storage point.The gain grows with SRAM share and diminishes when larger HBM capacity lets recency approach the hit-rate ceiling; ancillary parameter changes shift mean AMAT gain by less than 0.6 percentage points around the 19.6% baseline.Source: E. Robustness Across Capacity… · pp. 8–9, Table III · p. 9

F. Hazard LUT Fairness and Online Convergence

G. Production-Stack Validation with vLLM

The vLLM integration tests SPEAR against LRU using runtime-observable metadata and finds latency and hit-rate gains in representative cache-bound and capacity-stressed cases. It also identifies regimes where software timing or GPU queuing limits the gains; the hardware evaluation then assesses the dedicated scheduling core.

G. Production-Stack Validation with vLLM

  • In the cache-bound vLLM case, SPEAR cuts mean TTFT by 35% and tail end-to-end latency by 72%.The test pairs a small, fast-decode model with a tight tool-call schedule; the capacity-stressed case gains 2.5 pp hit rate and reduces mean TTFT by 6%. The native 30B case gains 1.6 pp hit rate.Source: G. Production-Stack Validation with vLLM · p. 10
  • The software patch can lose 6.6 pp hit rate when tight tool gaps fall below decode latency, while GPU-bound workloads may show no latency gain.The vLLM integration tests SPEAR only: its block cache has no tier-placement API for evaluating TIDE.Source: G. Production-Stack Validation with vLLM · p. 10

VI. HARDWARE EVALUATION

  • The hardware evaluation assesses the 28-nm scheduling core through synthesis, efficiency, timeliness, fidelity, structural necessity, and a board-level demonstration.The evaluation argues that the same policy cannot be realized in software or as two independent IPs without reintroducing documented failure modes.Source: §VI · p. 10

A. Locus Against Datapath KV Hardware

UNISON’s near-memory scheduler ranks shared sessions, unlike existing KV hardware focused on a single request’s token or block metrics. Its 64-session core has low reported area and power, with decision costs small relative to the managed KV hierarchy.

A. Locus Against Datapath KV Hardware

B. Synthesis Methodology

  • Synthesis uses a commercial 28-nm CMOS library at 150 MHz, with vectorless power estimation and bit-exact validation on the hazard-sensitive GAIA/Gemma4 Edge-Tight trace.Timeliness analysis uses all three GAIA families because only those traces have recorded tool-gap annotations.Source: B. Synthesis Methodology · p. 11

C. Area, Power, and Timing

  • The 64-session design occupies 0.169 mm2, uses 13.6 mW, and meets 150 MHz.The reported estimate is 153.4 MHz Fmax with 0.01 ns positive setup slack; the critical path traverses the first TIDE divider stage.Source: C. Area, Power, and Timing · p. 11, Table VII · p. 11

D. Decision Efficiency and Control-Plane Overhead

  • The 0.169 mm2 controller manages a 10 K to 60 K token pool and delivers a 34.8% mean AMAT reduction.At 13.6 mW and a 2.00 µs mean decision latency, each eviction or migration decision consumes approximately 27.2 pJ. Controller area scales with session capacity rather than model dimensionality or sequence length.Source: D. Decision Efficiency and… · p. 11

E. Decision Timeliness

UNISON’s hardware decision latency is small relative to observed waits, its implementation closely matches floating-point ranking, and the structural analysis ties its operation to a unified near-memory design.

E. Decision Timeliness

  • At N=64 and 150 MHz, worst-case decision latency was 0.31% of the minimum observed gap, while millisecond software polling can invert the gap signal on short waits.A ten-fold frequency reduction to 15 MHz yields 31 µs decision latency, still within the millisecond-scale minimum gap.Source: E. Decision Timeliness · pp. 11–12

F. Implementation Fidelity

  • Fixed-point decisions had 0.55% top-1 disagreement, but mean full-trace hit-rate deviation was 0.019 pp and did not accumulate into trajectory-level drift.The residual was smaller than the inter-policy gap between UNISON and CacheScout on every dataset.Source: F. Implementation Fidelity · p. 12

G. Structural Necessity of the Unified Near-Memory Design

  • Software timestamping inverts short-gap signals, split eviction/migration IPs lose up to 5.4 pp hit rate, and the unified design combines event visibility, shared state, and deterministic completion.The analysis presents these constraints as requiring a near-memory control plane rather than independent IPs or software.Source: G. Structural Necessity of the Unified… · p. 12

H. Board-Level Demonstration

  • The Xilinx Zynq-7020 prototype replays 1,937 events and 845 decisions; its netlist uses 43,415 LUTs, 20,938 flip-flops, and 20 DSP48E1 blocks.The netlist reports 81.6% slice utilization, 19.7% flip-flops, 9.1% DSP48E1 utilization, and no block RAM.Source: Figure 8 · p. 12, H. Board-Level Demonstration · p. 12

I. Physical Placement Considerations

  • The controller need not share the HBM PHY clock: a sideband memory-controller event bus provides the needed event stream without a shared clock or high-bandwidth data interface.The placement discussion identifies event access, rather than clock-domain co-location, as essential.Source: §I · p. 12

VII. CONCLUSION

  • UNISON was the best non-oracle entry on hit rate and AMAT on every trace, raising hit rate by 0.3% to 23.1% and reducing AMAT by 22% to 51%.On long-horizon traces, it also lowered serving TTFT by 58% to 89% when extra prefill sat on the critical path; the 28-nm core occupies 0.169 mm2 at 13.6 mW and 150 MHz for 64 sessions.Source: §VII · pp. 12–13

Related papers

Loading 2609.09643v1…