PaperlayerView exact revision on arXiv

Source-linked AI summary

PACE: Perceived-Latency-Aware Cascading Service Routing and Filler Control for QoE-Efficient Retrieval-Augmented Dialogue Serving

Lin Huang, Yujuan Tan, Weisheng Li, Lixiang Zeng, Kun Yang, Suihan Xiao

arXiv:2609.10372v1cs.CVcs.AIcs.RO

TL;DR

PACE addresses perceived response delay in retrieval-augmented dialogue by jointly controlling answer-source routing, waiting-window fillers, and volatile-cache admission. It deploys a load-adaptive cascade, path–filler controller, and volatility-aware admission in a humanoid-robot sales service. The system reduces latency and stale answers while preserving the static baseline under stationary conditions, with limitations in domain, language, baselines, and heuristic guarantees.

  • Problem

    Retrieval-augmented dialogue needs to reduce perceived response delay while avoiding stale cache answers and filler–answer contradictions in time-sensitive sales interactions.

  • Method

    PACE jointly controls heterogeneous answer-source routing, filler emission and duration, and volatility-aware cache admission, with a gate that switches between adaptive and static operation.

  • Results

    PACE halves pure-LLM PTFR P95 at c16 (0.29 vs 0.53 s), cuts filler calls by 94% with zero conflict, and reduces stale answers from 86% to 0%.

  • Takeaways & Limitations

    PACE provides a coordinated serving framework that combines latency, quality, cost, filler safety, and freshness controls in a deployed dialogue service.

  • Takeaways & Limitations

    The evaluation centers on one vertical, uses Chinese-language sales artifacts, omits model-tier cascade baselines, and provides heuristic rather than regret-bound guarantees.

Abstract

from arXiv · show

We present the PACE, a framework for retrieval-augmented dialogue serving that formalizes Perceived Time-to-First-Response (PTFR) as a QoE objective and minimizes it under quality/cost constraints. Unlike prior work on cascaded routing, semantic caching, or adaptive retrieval, PACE jointly controls which answer source composes the response and what fills the waiting window. Deployed on a humanoid-robot sales service, it combines three mechanisms: a load-adaptive cascading router, a joint path-filler controller, and volatility-aware cache admission. On 75k CarQA requests, the cascade halves pure-LLM PTFR at P95 (0.29 vs 0.53s at c16). The adaptive controller reaches 0.41s P95, outperforming RAG by 2.4 times at high load with equal quality. The filler controller cuts calls by 94% with zero conflict. Volatility-aware admission reduces stale answers from 86% to 0%. A gating rule ensures the controller never worse than the baseline, with exposure bounded by one hold period. This is the first quantification of filler-answer conflict risk in deployed services.

I. INTRODUCTION

PACE frames retrieval-augmented dialogue serving around perceived time to the first substantive response, subject to quality and cost constraints. It combines adaptive cascade routing, joint filler control, and freshness-aware caching in a deployed humanoid-robot sales service.

  • Motivation: PACE asks how to minimize perceived time to the first substantive response while satisfying answer-quality and cost constraints.It treats the serving stack as a service-composition problem involving retrieval and a cloud-hosted LLM.
  • Motivation: The cascade spans semantic caching, retrieval direct return, and full LLM generation, trading latency, quality, and cost across answer sources.The corresponding first-token regimes are near-zero for L0, 0.2–0.5 s for L1, and 1–4 s for L2.
  • Contribution: PACE jointly controls which answer source responds and what the user sees while waiting, addressing an intersection left uncovered by prior research threads.The paper positions this combination against QoS composition, cascade routing, semantic caching, freshness-aware caching, and filler research.
  • Contribution: Its PTFR formalization separates the first informative token from filler timing and instruments each request’s path, stage latencies, routing, filler, and volatility decisions.The decomposition is PTFR = temb + troute + tpath, with an explicit filler-coverage account.
  • Contribution: The load-adaptive router coordinates cache and direct-return thresholds using online load signals, while the path–filler controller jointly decides whether and how long to emit a filler.The router uses TTFT and arrival rate; the filler controller can skip generation when instant replies are likely.
  • Contribution: Volatility-aware admission distinguishes time-sensitive queries from stable ones, limiting stale cache responses without requiring change-detection infrastructure.The rule targets prices, promotions, inventory, and similar volatile sales queries.

B. LLM serving and cascaded routing

PACE extends cascaded serving from objective latency and cost control to perceived first-response latency. Its formulation couples routing and filler decisions because each changes the other’s value under a quality–delay–cost objective.

  • Research positioning: Prior serving methods reduce generation latency or route among models, but do not jointly choose heterogeneous answer sources and waiting-window content.PACE differs from systems that optimize kernel, scheduler, cache, retrieval, or model-tier decisions in isolation.
  • Problem formulation: PACE defines perceived latency as the earlier of filler first-frame time and informative first-token time when a filler is shown.The filler covers the waiting window when tf ≤ ti, making perceived latency equal to tf.
  • Problem formulation: The serving objective minimizes perceived latency subject to judged answer quality and per-request token and small-model cost bounds.The constraint pair creates a quality–delay–cost trade-off rather than pure latency minimization.
  • Problem formulation: The operating environment is non-stationary because conversation bursts and upstream TTFT variation shift load, making fixed thresholds unreliable.PACE therefore uses an online sequential policy rather than an offline-tuned static configuration.
  • Problem formulation: Routing and filling are coupled because an instant path makes a filler worthless, while shifting traffic from L2 to L1 removes both latency tail and filler opportunity.The policy therefore chooses routing thresholds together with filler launch and waiting budget.
  • Policy design: PACE presents a deployable heuristic policy with first-order model motivation, stability guarantees for online estimates, and empirical evaluation of approximation error.The paper does not claim a closed-form solution to the constrained program.

B. System architecture

PACE executes each request through a single-embedding, three-level cascade while speculative filler generation runs in parallel with uncertain routing. Instrumented timing exposes how each path contributes to perceived response time.

  • Filler execution: The filler is speculative: it launches before the routing outcome is known and is cancelled if an instant path answers.This design prevents filler generation from delaying an instant answer.
  • Request pipeline: Each request is embedded once and evaluated through L0 semantic cache, L1 retrieval direct return, or L2 LLM streaming.The same vector serves cache lookup and retrieval search, while L0 hits skip embedding and retrieval.
  • Request pipeline: The router sets cache and direct-return thresholds from online signals, while the filler controller decides whether a small model occupies the waiting window and for how long.Volatility-aware admission separately guards time-sensitive cache entries.
  • Timing anatomy: On L0 and L1 the answer itself is the first frame; on L2 the filler and substantive stream run in parallel, with the perceived first response arriving at min(tf, ti).The filler frame can appear while the substantive answer continues behind it.

C. Load-adaptive cascading router

PACE combines TTFT and arrival rate into a load index that jointly adjusts cascade thresholds, while gating adaptation during changing regimes and reverting to a static operating point when stable.

  • Load-index construction: The router combines EWMA-smoothed LLM TTFT with arrival rate into a normalized load index weighted 0.7 for TTFT and 0.3 for rate.TTFT captures perceived-latency degradation, while arrival rate provides a leading burst indicator.
  • Threshold adaptation: Thresholds for semantic-cache and retrieval-direct paths decrease affinely as load rises, using operator-chosen intervals to trade quality for bounded PTFR.At λ=0, θcache=0.98; at λ=1, looser thresholds admit paraphrases and moderate-confidence retrievals.
  • Controller properties: The affine controller is monotone, memoryless, and requires no offline training or calibration corpus, enabling transfer across domains or embedding models.The EWMA tracks step changes within roughly five observations under α=0.3.
  • Endogeneity and stability: Because TTFT is updated only by L2 completions, loose thresholds can reduce samples, but arrival counting and residual L2 misses prevent autonomous oscillation.If starvation occurs, the index freezes near the last observed regime rather than entering a limit cycle.
  • Steady-state gating: A gate switches between ADAPTIVE and STATIC modes using rate, load-fluctuation, and latency-pressure triggers over 60-second statistics.The gate addresses changing regimes while allowing stable operation at a static point.

D. Joint path–filler controller

PACE jointly decides whether to show a filler and how long to wait, while controlling stale cache admissions for volatile sales queries that can conflict with visible waiting messages.

  • Launch decision: The filler controller skips generation when recent instant-path probability P(inst) exceeds 0.7, because the answer is expected to arrive no later than the filler.P(inst) is estimated over the last N=32 routing decisions with a cold-start prior of 0.3.
  • Waiting budget: The controller uses a path-dependent waiting budget instead of the production system’s fixed 0.9-second deadline.The budget accounts for TTFT and elapsed embedding and retrieval time, with β=0.8, Bmin=0.2 s, and Bmax=1.2 s.
  • Conflict control: A filler is constrained to avoid factual anticipation, and filler–answer conflict rate measures visible fillers that disagree with the subsequent answer.Prompt rules forbid facts and “let me check” phrasing; filtering rejects outputs longer than 30 characters or containing search-announcing tokens.
  • Volatility motivation: Volatility-aware admission distinguishes stable from volatile queries because prices, promotions, inventory, and today-type questions change on hourly-to-daily timescales.A similarity threshold alone cannot prevent a latency-optimized cache from serving outdated discounts.
  • Cache admission: A lexical volatility prior assigns stable or volatile labels, then applies differentiated admission, TTL, and eviction rules without knowledge-base monitoring.Volatile entries receive a short TTL or rejection, while stable entries use LRU/LFU eviction.
  • Generality: The volatility interface is domain-independent, but the classifier is domain-specific and should prioritize high recall for volatile queries.Stable queries mislabeled volatile incur only extra regeneration.

F. Formal properties of the heuristic rules

PACE’s heuristic rules receive local optimality, stability, and gating interpretations, but the paper limits these results to first-order finite-sample reasoning rather than global regret guarantees.

  • Threshold optimality: The affine threshold schedule is first-order optimal around a calibrated operating point and uniquely matches the KKT condition at both boundary regimes.Its endpoints represent quality-first idle operation and saturation.
  • Load-index weighting: The deployed 0.7/0.3 load-index weighting is inverse-variance optimal when the rate estimate is about 7/3 noisier than the EWMA.Interior weights preserve boundary fixed points and monotonicity while changing only transient behavior.
  • Estimator stability: The EWMA is a contraction, and its initialization error decays geometrically as (1−α)^n.Under bounded drift δ, the steady-state tracking bias is at most δ(1−α)/α; clipping also prevents autonomous threshold oscillation.
  • Filler decision: The filler launch rule is Bayes-optimal for cost ratio c/(c+u)=0.3, with measured filler-call reduction identified as its predicted behavior in instant-path-dominated traffic.The rule launches when bp ≤ 1−c/(c+u), where bp=P(inst).
  • Staleness bound: TTL admission bounds volatile-answer staleness by Pr[stale] ≤ 1−e^(-reτ) ≤ reτ, allowing a target freshness level through TTL selection.The regeneration cost is borne only by the volatile fraction.
  • Gating equivalence: Under stationary load without triggers, gated PACE is path-identical to PACE-static, with residual regime-change exposure bounded by at most H+1 windows.The result gives zero steady-state regret relative to a hand-tuned static configuration.
  • Scope: The formal results are local and finite-sample, and calibrated constants come from deployment practice rather than being derived from the stylized models.Porting requires re-estimating measurable quantities; replacing the affine law with a learned controller remains future work.

G. Implementation

PACE is deployed as an OpenAI-compatible service and evaluated through fixed, instrumented campaigns spanning CarQA, volatile-query, paraphrase, and cross-domain datasets with end-to-end latency, quality, cost, and path measurements.

  • Service stack: PACE exposes an OpenAI-compatible /v1/chat/completions endpoint, requiring only a base-URL change for the dialogue manager.The service uses a DeepSeek L2 model, a capped Qwen-class filler, Qdrant retrieval, and circuit-breaker protections.
  • Measurement campaigns: The main campaign contains 75,000 instrumented requests across ten arms, while a separate 6,000-request DuReader campaign reuses the frozen harness configuration.Protocol decisions were fixed independently of outcomes.
  • Datasets: The evaluation includes CarQA-3k, CarQA-Para, CarQA-Volatile, and DuReader-3k, with scripted price-change events isolating stale-answer behavior.The datasets derive from production robot conversations and include a cross-domain transfer check.
  • Systems and ablations: End-to-end comparisons cover Pure LLM, Standard RAG, GPTCache, PACE-static, PACE, and mechanism-specific ablations.Ablations separately vary threshold adaptation, filler control, volatility handling, and cascade depth while controlling state contamination.
  • Metrics: Latency evaluation reports PTFR, filler first-frame time, perceived latency, and completion time from per-request trajectories at specified percentiles.PTFR is primary because perceived-latency masking assumptions are separately audited through PLα.
  • Quality and cost: Quality uses blinded LLM judging calibrated against 300 double-blind human-scored items, while cost reporting includes tokens, filler calls, path shares, stale rejects, and load trajectories.Judge-bias controls include model-family separation, randomized presentation, and rubric examples.

D. Load-sweep protocol

The load-sweep protocol evaluates PACE and comparison systems across controlled concurrency levels, with PTFR distributions, ablations, and transition behavior examined under stationary and changing load. The experiments test whether adaptive routing tracks or improves on a hand-tuned operating point without per-deployment tuning.

  • Protocol: The experiment varies offered concurrency c ∈ {1, 4, 8, 16, 32} using held-out CarQA requests, repeated three times with matched statistical comparisons.The pre-registered primary endpoint is PTFR P95 at c=16, comparing PACE-full with PACE-static.
  • Stationary load: PACE-static halves pure-LLM PTFR P95 at c=16, from 0.53 s to 0.29 s, while PACE-full reaches 0.41 s without per-deployment tuning.The table reports means with 95% confidence intervals over three seeds.
  • Stationary load: At c=32, both cascades outperform standard blocking RAG by more than 2.4×, while cascade ordering differs between cold-cache c=1 and warm-cache c≥4 regimes.At c=1, both cascades lose to pure LLM because embedding and cache checking add overhead before the cache warms.
  • Distributional evaluation: Figure 3 measures PTFR P95 across offered concurrency, and Figure 4 shows PTFR CDFs at c=16 and c=32 against the deployment’s 0.5 s responsiveness target.Standard RAG’s high-load tail extends beyond 1 s, whereas both cascades concentrate most mass below 0.4 s.
  • Non-stationary stress test: During the non-stationary ramp, PACE-full is never worse than PACE-static and is descriptively better in every transition phase reported.PTFR P95 differences are 0.20, 0.11, 0.21, and 0.40 s in phases P1–P4, with a 30% margin during ramp-down.
  • Scope of claims: The tested loads do not produce remote-backend queueing breakdown, so the results support tracking without tuning rather than catastrophe avoidance.The adaptive arm matches or beats the hand-tuned point during every induced load transition, while removing the assumption that load remains at the tuning regime.

G. Volatility evaluation

The volatility evaluation shows that cache admission controls stale answers by trading volatile-query hits for bounded latency, while the broader controller behavior remains load- and path-dependent. A frozen cross-domain replay reproduces the latency relationship without tuning.

  • Router behavior: The load index rises with concurrency, and PACE’s L2 share contracts as load grows while PACE-static’s path mix remains essentially unchanged.Measured thresholds track their designed affine schedules, providing the mechanism for regime-dependent ordering.
  • Filler control: The adaptive filler controller issues 94% fewer calls than the fixed-budget arm, with 0% judged conflict versus 2.6% for the fixed arm.Launches occur almost exclusively when P(inst) is at or below the 0.7 rule boundary.
  • Volatility-aware admission: 86.0% stale rate accompanies 96.6% volatile hit rate when volatility-aware admission is disabled.Denial trades a moderate latency increase for zero staleness.
  • Volatility-aware admission: Zero stale answers result when volatile entries are denied, while TTL policies evict entries after expiry and route queries to L2.TTL converts staleness into a bounded waiting period rather than eliminating volatile hits entirely.
  • Cross-domain transfer: On the frozen DuReader-3k transfer, PACE-full matches the static operating point at c=16 (0.288 vs. 0.271 s), with both 2.5× faster than the pure-LLM floor.The latency ordering reproduces without retuning under the same deployed configuration.

V. DISCUSSION

The discussion frames PACE as an operational QoE system whose gate preserves the static baseline in stationary regimes while adapting during transitions. It also identifies deployment boundaries, implementation requirements, and extensions beyond dialogue.

  • Limitations: The evaluation centers on one vertical, uses Chinese-language sales-register artifacts, omits model-tier cascade baselines, and relies on heuristic controllers rather than regret bounds.The authors identify multilingual replication and broader baseline evaluation as open boundaries.
  • Operational implications: A stale operating point requires retuning after backend, prompt, or corpus changes, whereas the engaged gate reaches the static point without human intervention.The paper characterizes zero tuning cost as the operational property delivered by the system.
  • Service management: PTFR P95 is proposed as an SLA-style service indicator, with threshold intervals as a quality envelope and the kill switch as a compliance mechanism.These artifacts translate the framework into service-level management instruments.
  • Deployment guidance: The components are independently adoptable, but routing and filler control require PTFR instrumentation, threshold intervals, and deployment-specific load anchors.Volatility admission additionally requires a query-side rule and TTL field.
  • Embodiment and future work: The formulation extends naturally to embodied channels by measuring the first perceivable response and coupling routing with multimodal filler onset.The dialogue evaluation does not itself constitute full embodiment.
  • Operational implications: PACE matches or beats the hand-tuned baseline through induced load transitions and never trails it beyond one hold period.The conclusion presents this guarantee together with zero tuning cost.

I. NOTATION

The supplementary notation and gate-replay analysis describe how PACE switches between adaptive and static operation under changing versus stationary load. The replay validates the intended mode occupancy and stationary equivalence.

  • Gate validation: Stationary jitter remains below the gate’s trigger thresholds, supporting stable closure under constant offered load.Measured rate CV is ≤0.16 at the 99th percentile, versus the fluctuation trigger Fw>0.25.
  • Adaptive mode: Under continuously non-stationary load, the gate stays ADAPTIVE through every transition, with P2–P5 receiving 100% adaptive-mode traffic.The gate opens at cold start and closes only on the final window of the initial c=1 plateau.
  • Static mode: Under stationary load, the gate closes after the cold-start hold and routes a mean 75% of requests in STATIC mode.The static share ranges from 40–80% and tends toward 100% as stationary periods lengthen.
  • Embodiment: The formulation accommodates additional embodied channels by extending PTFR from first-token time to first perceivable response onset.It also couples filler launch to routing and adds an explicit conflict metric for embodied interaction.
  • Transfer replay: In DuReader-3k, frozen deployment settings reproduce the latency ordering without tuning: at c=16 PACE-full is 0.288 versus 0.271 s for the static point.Both are reported as 2.5× faster than the pure-LLM floor.
  • Transfer replay: The automotive volatility lexicon fires on only 3.2% of open-domain requests, with 100% routed through the instant direct path and no measured latency penalty.The prior gates cache admission rather than retrieval service.

C. Where the domain specificity actually lives

The transfer analysis localizes domain specificity in the prompt and volatility lexicon rather than the cascade architecture. Supporting propositions formalize the router, filler, TTL, and gate behaviors used by PACE.

  • Domain specificity: PACE-static scores 4.94 and PACE-full 4.72 on open-domain quality, while L2-only RAG and GPTCache score 2.21 and 2.32.The frozen automotive persona prompt penalizes L2-only systems, while PACE’s fast levels bypass the LLM.
  • Domain specificity: The pure-LLM arm scores 3.57, and PACE-full’s lower score reflects its 0–11% L2 share under the frozen persona prompt.The architectural reading is that domain-specific prompting is confined to the small L2 traffic share.
  • Router properties: The router’s monotone loosening proposition states that the interior optimum threshold decreases as load increases.This follows from the KKT condition and an increasing latency-to-quality exchange rate.
  • Router properties: The deployed affine threshold schedule is first-order optimal around a calibrated operating point and uniquely matches both boundary solutions to first order.The schedule is the secant through the two calibrated boundary solutions.
  • Filler properties: The filler launch rule is Bayes-optimal for the path-mixture statistic, launching when bp ≤1 −c/(c + u).The threshold balances unnecessary-launch loss against uncovered-slow-path loss.
  • Cache properties: A TTL τ bounds stale-answer probability by Pr[stale] ≤1 −e^(-reτ) ≤reτ when knowledge-change events arrive at rate re.The bound is maximized at the TTL age.
  • Gate property: After H trigger-free windows, the gate pins thresholds at (0.95, 0.75), making stationary routing distributions identical to PACE-static.Residual regime-change exposure is bounded by the detection-plus-hold lag of at most H+1 windows.

VI. WATERFALL DECOMPOSITION OF THE MAIN GAIN

PACE’s PTFR P95 gain is decomposed across cascading, adaptive thresholds, filler control, and volatility-aware admission, with cascading contributing the largest share. Across systems at c=16, RAG-backed designs maintain similar quality while cascades differentiate themselves primarily through lower latency.

  • Waterfall decomposition: The L0/L1 cascade contributes the largest share of PACE-full’s PTFR P95 gain over pure LLM at c=16.The decomposition also separates adaptive thresholds, adaptive filler, volatility admission, and residual contribution.
  • Waterfall decomposition: Removing the L1 direct-return level is the single most damaging ablation.This supports the role of mid-confidence retrievals, which carry a large fraction of traffic at near-cache latency.
  • Quality–latency trade-off: At c=16, RAG-backed systems cluster within a quality band of 4.65–4.81/5, whereas pure LLM reaches 2.84/5.The deployment’s answers depend on retrieval-grounded facts.
  • Quality–latency trade-off: Within the RAG-backed quality band, cascades occupy the faster lower-left region, and latency varies more than quality across serving designs.The figure describes latency, rather than quality, as the primary differentiator once systems remain in the narrow quality band.
  • Volatility evaluation: Without admission control, per-bucket stale rates remain approximately 86–91% regardless of event age.The cache cannot detect that a price-changing event has occurred.
Loading 2609.10372v1…