Source-linked AI summary
Beyond Static RAG: An Adaptive, Tri-Metric Routing Framework for Efficient Long-Context Inference on Commodity GPUs
Saipraveen Vabbilisetty, Ajay Kumar Boddepalli, Deep Narayan Mishra, Shashank Kapadia, Haoan Wang, Anupriya Sharma
TL;DR
Long-context RAG on commodity GPUs faces a Compression Paradox: compression can add latency and memory contention, while omitting it can cause OOM failures. The paper introduces a deterministic Tri-Metric Router that selects Raw, Neural, or Lexical pipelines using hardware- and content-derived signals, achieving 0% OOM, 88.5 ± 4.4% OOD oracle alignment, and 49.3 ± 1.4% Combined F1 while improving over Always-Lexical by 5.2 points.
Problem
Long-context RAG on 16 GB commodity GPUs can suffer compression-induced latency and VRAM contention, while raw processing can trigger OOM failures.
Method
The Tri-Metric Router deterministically selects Raw, Neural, or Lexical pipelines using length, syntactic density, type-token ratio, VRAM headroom, and calibrated latency crossover thresholds.
Results
On out-of-distribution holdouts, the router achieves 0% OOM, 88.5 ± 4.4% oracle alignment, and 49.3 ± 1.4% Combined F1, gaining 5.2 points over Always-Lexical.
Takeaways & Limitations
The results support hardware-physical routing as a way to balance latency, memory safety, and answer fidelity without additional training or VRAM.
Takeaways & Limitations
The 4,332-word crossover is an operating-point estimate for the tested T4 configuration, and transfer to other hardware requires recalibration and empirical validation.
Abstract
from arXiv · showhide
Deploying retrieval-augmented generation (RAG) on commodity GPUs such as the NVIDIA T4 (16 GB VRAM) exposes a practical failure mode we call the Compression Paradox: neural prompt compression can add key-value (KV) cache contention and preprocessing latency that outweigh generation-time savings, while skipping compression can cause out-of-memory (OOM) failures on long contexts. We identify two distinct failure mechanisms when a vLLM-served LLM and a PyTorch-based compressor are co-deployed under tight memory budgets, and introduce the Tri-Metric Router, a deterministic, training-free policy that selects among Raw, Neural (LLMLingua-2), and Lexical (BM25) pipelines. The router uses three CPU-side signals: spatial complexity ($L$), syntactic density ($ρ_{key}$), and type-token ratio (TTR). Unlike prior semantic-only adaptation, our dispatch signal is hardware-physical, based on VRAM headroom and a latency crossover point. Thresholds are calibrated from profiling on LongBench qasper, yielding an operating crossover near 4,332 words on T4; our contribution is this calibration methodology rather than a hardware-specific constant. On out-of-distribution holdouts, the method achieves 0% OOM failures, 88.5 $\pm$ 4.4% oracle alignment, and 49.3% Combined F1, improving over always-on lexical compression by 5.2 points without additional VRAM or training cost.
1. Introduction
The paper frames long-context RAG on 16 GB commodity GPUs as a hardware-constrained deployment problem and introduces deterministic routing across compression pipelines. Its contribution is hardware-physical adaptation that targets compression-related failures without learned routing components.
- The Compression Paradox arises when neural compression overhead and VRAM contention can make inference slower and less reliable than raw passthrough on long contexts.The paper identifies latency inversion and distinct memory failures under constrained co-deployment.
- Existing adaptive-RAG methods adjust retrieval while treating post-retrieval compression and serving as fixed, hardware-agnostic pipelines.
- The Tri-Metric Router is deterministic, training-free, and auxiliary-model-free at dispatch time, selecting among hardware-safe execution paths.
- The router repurposes spatial complexity, syntactic density, and type-token ratio as protective heuristics for hardware-constrained compression selection.
- The framework co-locates vLLM and a PyTorch neural compressor through sequential initialization and validates a stable T4 utilization range of u ∈[0.50, 0.80].
2. Related Work
Prior work supplies neural, lexical, cache-management, and adaptive-routing techniques, but the paper positions its contribution as routing compression mechanisms within a fixed LLM serving stack according to hardware constraints. This distinguishes hardware-aware execution routing from semantic or model-economic adaptation.
- Neural prompt compression: LLMLingua-2 provides the neural compression baseline, while RECOMP represents query-conditioned abstractive and extractive compression with additional encoder–decoder overhead.
- Lexical compression: BM25 offers CPU-only, deterministic-length lexical compression and serves as an OOM-safe fallback for inputs that threaten neural compression.
- KV-cache management and efficient serving: PagedAttention, AWQ, KV-cache eviction, streaming, and KV quantization reduce or manage memory pressure, whereas this work routes among compression pipelines.
- Cost-aware cascading: Unlike cost-aware cascading, the router holds the LLM fixed and dispatches across Raw, Neural, and Lexical mechanisms within one GPU serving stack.
- Hardware-aware vs. semantic routing: Existing Adaptive-RAG and model-routing systems primarily use semantic difficulty or retrieval necessity, while this work makes VRAM contention and latency crossover first-class dispatch signals.
3. Methodology
The methodology formulates routing as minimizing expected end-to-end latency under a hard VRAM constraint, then derives a deterministic policy from profiling and hardware-safe co-residency calibration. The policy uses length, syntactic density, and lexical diversity to choose Raw, Neural, or Lexical processing.
- 3.1. Problem Formulation: The router minimizes expected end-to-end latency, combining compression latency with generation latency, subject to total memory not exceeding the T4 capacity.The theoretical objective is approximated empirically because cross-process contention makes latency non-deterministic.
- 3.2. VRAM Partitioning Framework: The T4 deployment reserves vLLM and a static neural-compressor slice, leaving residual headroom for dynamic PyTorch allocations.
- 3.2. VRAM Partitioning Framework: The strict initialization hierarchy loads vLLM first and then LLMLingua-2, preventing uncontrolled allocator contention that caused OOM in all tested trials.
- 3.2. VRAM Partitioning Framework: The empirically safe Goldilocks Zone is u ∈[0.50, 0.80], bounded by PyTorch starvation at high utilization and KV-cache exhaustion at low utilization.
- Signal definitions: The router defines L as word-count spatial complexity, ρkey as keyword-token density, and TTR as vocabulary diversity used to identify compressibility and structural risk.
- Calibrated thresholds: Profiling calibrates θk = 0.05, θt = 0.45, Llow = 1,500 words, and an operating crossover L∗ = 4,332 words with a within-sample bootstrap interval of ±185 words.The paper emphasizes that the scalar crossover is configuration-specific and that calibration methodology, not the constant, is the contribution.
- Decision policy: Algorithm 1 routes short inputs to Raw, medium low-density inputs to Neural, long inputs through BM25 plus a 4,096-token ceiling, and dense medium inputs back to Raw.The Raw fallback protects code- and formula-dense content from neural removal of load-bearing syntax.
4. Experimental Setup
The evaluation uses a 100-document LongBench qasper sweep on a single 16 GB NVIDIA T4, comparing fixed Raw, Neural, and Lexical baselines with the Tri-Metric Router. It measures crash rate, latency, oracle alignment, and token-level answer quality while accounting for crash-induced zero scores.
- Models and baselines: The study evaluates Llama-3-8B-Instruct-AWQ with LLMLingua-2 and BM25, comparing Always-Raw, Always-Neural, Always-Lexical, and the Tri-Metric Router.
- Evaluation protocol: The 100-document qasper sweep is stratified into Short, Medium, and Long bands using Llow and L∗, with 34, 45, and 21 documents respectively.
- Metrics: Evaluation reports OOM rate, mean end-to-end latency, oracle alignment, and official LongBench token-level F1 on completed samples.Combined F1 assigns zero to crashed documents, while survival-biased values are shown separately.
5. Results
The Tri-Metric Router avoids distinct baseline failure modes while preserving quality across in-distribution and out-of-distribution evaluations. Its headline OOD operating point combines 0% OOM with 88.5% oracle alignment and 49.3% Combined F1, while remaining subject to stated safety and evaluation limitations.
- 5.1. System-Level Comparison: Always-Raw crashed on 65% of long-context samples through CacheEngine block-pool exhaustion, whereas Always-Neural crashed on 18.3% through cross-process PyTorch CUDA OOM.The mechanisms differ: raw inference exhausts vLLM’s page pool, while the neural encoder is starved by vLLM’s rigid reservation.
- 5.1. System-Level Comparison: 51.7 ± 1.2% Combined F1 across all 100 documents exceeded Always-Lexical’s 47.8 ± 1.5% while maintaining 0% OOM.The router also reported 7.91 ± 0.3 s mean latency; crash-aware Combined F1 assigns zero to crashed documents.
- 5.1. System-Level Comparison: Removing the syntactic-density and TTR guards reduced OOD Combined F1 by 1.5 points, from 49.3% to 47.8%, when all medium documents were routed to Neural.The probe used N=50 and should be distinguished from the local N=6 guard-triggering ablation, which was underpowered and not statistically significant.
- 5.2. Out-of-Distribution Routing Agreement: 49.3 ± 1.4% OOD Combined F1 and 88.5 ± 4.4% oracle alignment were achieved with 0% OOM, outperforming Always-Lexical by 5.2 points.The comparison used frozen thresholds on 50 multifieldqa en documents; the gain was significant under a 10,000-resample bootstrap permutation test (p = 0.014).
- 5.3. Per-Band Routing Breakdown: At 50% compression, LLMLingua-2 produced 8.63 s net latency on a 3,100-word input, while end-to-end Short, Medium, and Long latencies were 2.11 s, 8.63 s, and 4.72 s.The reported setting was batch size B=1 on a 16 GB T4; batched serving was not evaluated, and the neural band is expected to narrow or disappear at B ≥4.
6. Limitations, Calibration Transfer, and Future Work
The paper presents a transferable calibration procedure for hardware-aware routing while delimiting its hardware, model, benchmark, batch-size, and statistical scope. Future work targets cross-device validation, online adaptation, broader corpora, and integration with cache-management methods.
- Calibration Transfer: The router transfers through recalibration: profile each target GPU, fit the latency relationship, and solve for its operating crossover rather than reusing T4 thresholds.Only L* and Llow require hardware-specific profiling; the procedure uses at least 30 documents spanning 500–8,000 words.
- Limitations: L* = 4,332 words is a T4 operating-point estimate, with ±500–1,000 words of uncertainty beyond the within-sample bootstrap interval.The estimate derives from NLong = 21 documents, and the reported ±185-word interval reflects within-sample stability only.
- Limitations: The evaluation uses one LLM, one neural compressor, and LongBench English QA, leaving alternative models, benchmarks, and multilingual inputs untested.Specific L* values depend on compressor throughput and LLM generation speed, while the ρkey keyword set is validated only for English academic QA.
- Limitations: All experiments assume B = 1; at B = 4, Neural KV-cache headroom shrinks to ≈0.9 GB, compressing or eliminating the Neural routing band.Cache-management methods require instrumentation inside the attention loop and are proposed as future inner-loop additions.
- Future Work: Future work includes calibration on L4, A10G, and consumer GPUs, joint routing-surface optimization, online VRAM monitoring, broader ρkey coverage, and cache-management integration.Projected L* values for other hardware are estimates requiring empirical validation, and evaluation at B ∈ {2, 4} is planned.
Impact Statement
The work targets cost-efficient and reliable LLM inference on commodity accelerators, with no foreseeable negative societal consequences specific to the work beyond general LLM deployment.
- The work targets cost-efficient, reliable LLM inference on commodity accelerators, lowering the compute barrier for RAG deployment outside high-end environments.
A. Out-of-Distribution Evaluation Details
On unseen multifieldqa content, the evaluation exercises all routing bands and secondary signals, while preserving the router’s safety and quality advantages over fixed pipelines. The reported Combined F1 gap is statistically tested on the 50-document OOD split.
- Evaluation Design: The 50 OOD documents span Short, Medium, and Long bands, with 35 Medium/Long cases actively exercising secondary guards and BM25 fallback.This composition means the reported 88.5% agreement cannot be attributed solely to length-gate pass-through.
- OOD Safety: The router maintains 0% OOM failures across the OOD set, while Always-Raw and Always-Neural crash at 64% and 19%, respectively.The baseline crash rates remain within ±1 percentage point of their in-distribution rates.
- System-Level Comparison: The router reaches 49.3% Combined F1 versus 44.1% for Always-Lexical, a 5.2-point gap that is significant under paired bootstrap permutation testing.The test uses 10,000 resamples over the 50 OOD documents and yields p = 0.014.
- System-Level Comparison: Always-Neural’s survival-biased OOD F1 is 43.9%, below Always-Lexical’s 44.1%, reversing their in-distribution ordering.Survival-biased F1 is computed only over Always-Neural’s 81% non-crashed documents.
B. LLM / Agent Usage Disclosure
The paper states that LLM assistance was limited to writing refinement and boilerplate autocomplete, while technical claims, experiments, code, and findings remained human-authored.
- LLM assistance was limited to grammar, phrasing, clarity edits, and boilerplate autocomplete.The authors state that Claude assisted writing refinement and Gemini Pro assisted boilerplate only.
- No autonomous agent, code-execution agent, or multi-step LLM pipeline generated experiments, data, evaluations, or quantitative results.
- The authors identify the core contributions and empirical findings as original human-authored work.