PaperlayerView exact revision on arXiv

Source-linked AI summary

OUTLETS: Output-Length Prediction from Speculative Decoding Backbones

Weihuang Wen, Yingying Liu, Yichuan Liu, Wenqi Zeng, Li Zhou, Chumin Sun, Jie Sun, Tianshu Yu

arXiv:2609.01068v1cs.CL

TL;DR

LLM serving must handle highly variable output lengths, while existing predictors trade overhead against fidelity. OUTLETS reuses speculative-decoding backbone representations as a trajectory-aware length predictor, adding a lightweight regression head when those representations are already computed. Its predictions support standard scheduling in disaggregated serving, reducing short-request P99 latency and improving load distribution and throughput.

  • Problem

    Highly variable output lengths complicate resource provisioning and scheduling, while proxy predictors add overhead and hidden-state probes have limited fidelity.

  • Method

    OUTLETS repurposes speculative-decoding lookahead representations through a shared backbone and lightweight regression head for output-length prediction.

  • Results

    OUTLETS achieves lower prediction MAE than shallow target-state probes and provides scheduling signals that reduce HOL blocking, lower short-request P99 latency, and improve throughput.

  • Takeaways & Limitations

    Speculative drafters can support higher-level serving decisions in addition to token-level acceleration.

  • Takeaways & Limitations

    Standalone use requires counting the draft backbone cost, so lighter predictors may be preferable in extremely latency-sensitive settings.

Abstract

from arXiv · show

The heavy-tailed distribution of output lengths in Large Language Model (LLM) serving poses major challenges for resource provisioning and cluster scheduling. Although output-length prediction can mitigate these issues, existing approaches have key drawbacks: external proxy models add substantial latency and often have limited fidelity, whereas internal state-based methods are efficient but rely on shallow probes of current model states. We identify a structural connection between speculative decoding (SD) and length prediction: latent representations produced by the draft decoder in advanced frameworks (e.g., EAGLE-3) encode signals that are predictive of generation length. Building on this insight, we introduce OUTLETS (Output-Length Prediction from Speculative Decoding Backbones), which repurposes the speculative backbone as a trajectory-aware length predictor. When its draft representations are already computed for speculative decoding, OUTLETS adds only a lightweight regression head and achieves lower MAE than the evaluated methods. Under saturated disaggregated serving, OUTLETS predictions enable standard scheduling policies to prioritize shorter requests and distribute requests more evenly across decoding instances, reducing short-request P99 latency by 34.8%.

1 Introduction

LLM serving faces severe scheduling challenges from highly variable output lengths, while existing predictors trade accuracy against overhead. OUTLETS repurposes speculative-decoding representations for trajectory-aware length prediction and scheduling.

  • Motivation: Output lengths can vary by orders of magnitude, causing HOL blocking and ineffective agnostic dispatch under saturated serving.These effects degrade tail latency and make load balancing difficult.
  • Motivation: External proxy predictors add latency and memory overhead, while hidden-state MLP probes remain shallow and limited in fidelity.The methods therefore expose an overhead–accuracy trade-off.
  • Key insight: Speculative decoding and length prediction both model future sequence evolution, making draft-decoder lookahead states suitable for predicting termination.Unlike target-model states optimized for immediate next-token prediction, draft decoders roll representations forward.
  • Approach: OUTLETS uses a shared speculative backbone and adds a lightweight regression head when draft representations are already computed.The framework connects speculative decoding and length prediction at the representation level without introducing a new scheduling policy.
  • System impact: OUTLETS predictions provide scheduling signals that support standard load balancing and Shortest-Job-First scheduling in disaggregated serving.The system study reports reduced short-request tail latency and improved throughput.

2 Related Work

Prior work accelerates generation or predicts output length through external models, instructions, or internal states. OUTLETS connects speculative-decoding trajectory representations with length prediction.

  • Speculative decoding: Speculative decoding drafts multiple candidate tokens for parallel verification by the target model.Draft-and-verify methods reduce autoregressive generation bottlenecks but can add memory, compute, or synchronization costs.
  • Speculative decoding: EAGLE-family methods model future states or adapt draft structures, while adaptive-speculation methods optimize token-level lookahead within each iteration.These lines focus on token drafting rather than output-length prediction.
  • Length prediction: Instruction-based and proxy-based predictors can support scheduling or micro-batching but add prompt, latency, complexity, or resource costs.External predictors may also lack direct access to target-model runtime states, limiting fidelity.
  • Length prediction: Internal state-based methods attach lightweight predictors to target-model activations for continuous length or remaining-time estimation.These methods are model-native but generally probe states optimized for current-step prediction.
  • Connection to OUTLETS: OUTLETS bridges speculative decoding and output-length prediction by repurposing speculative lookahead representations for length prediction.The approach treats speculative backbones as signals for future generation trajectories, rather than only token drafting.

3 Preliminary

The paper formulates remaining-length prediction as estimating future generated tokens from information available during autoregressive decoding. It distinguishes static pre-decoding estimates from dynamic updates during generation.

  • Problem formulation: An inference request maps prompt x to an autoregressively generated sequence y of total length L until a stopping criterion is met.The stopping criterion may include producing an EOS token.
  • Problem formulation: At decoding step t, the true remaining length is r_t = L − t, and a predictor estimates it from the available model state.The predictor uses information available at the current step.
  • Static prediction: Static prediction occurs after prefill before output generation, where r_0 = L and the estimate supports initial resource planning.The paper identifies instance-level load balancing in disaggregated serving as an example.
  • Dynamic prediction: Dynamic prediction occurs during decoding as new output tokens are generated.Unlike static prediction, dynamic prediction updates with the realized generation trajectory.

4 Method

OUTLETS jointly optimizes speculative decoding and output-length prediction by sharing a speculative backbone and separating token drafting from length regression. It combines multi-depth feature fusion with a lightweight draft decoder and dual-head outputs to capture both local token dynamics and sequence-level structure.

  • Unified architecture: OUTLETS uses a shared speculative backbone to jointly generate draft tokens and predict output length with separate task-specific heads.The Draft Model Head supports speculative decoding, while the Length Regression Head estimates remaining generation cost.
  • Feature fusion: Multi-level target-model hidden states are concatenated and projected into the draft embedding space to provide richer contextual signals.The selected depths approximate shallow lexical, intermediate syntactic, and deeper semantic information.
  • Draft decoder: A lightweight Transformer draft decoder captures temporal dynamics and uses Gated Attention to preserve long-range termination signals.The method reports consistent length-prediction gains over standard Llama attention.
  • Length regression: The length head predicts z = log(1 + length) in log-space, producing an estimate of remaining generation cost at each step.Log-space regression is used because generation lengths are typically heavy-tailed.
  • Joint objective: OUTLETS trains end-to-end with speculative-decoding KL divergence, log-space length-prediction error, and L2 regularization.A fixed λ = 0.1 provided the best reported trade-off between stability and performance, while γ = 10^-5 regularized the length-prediction MLP.

5 Experiments

Experiments evaluate OUTLETS across models, datasets, prediction settings, overhead, and saturated disaggregated serving. OUTLETS achieves strong prediction accuracy with modest incremental runtime cost and improves throughput and short-request tail latency through length-aware scheduling.

  • Static prediction: OUTLETS outperforms proxy-based methods and the PIA-style prompting baseline across all evaluated datasets in static prediction.The result supports using hierarchical features from internal LLM states rather than shallow token-level representations from external proxies.
  • Dynamic prediction: 80.6 to 67.6: OUTLETS’ dynamic-prediction MAE decreases as decoding progresses on ShareGPT.The draft state captures trajectory information, enabling increasingly accurate updates during generation.
  • Prediction accuracy: OUTLETS consistently achieves the lowest MAE across model architectures and static and dynamic prediction scenarios.The evaluation reports MAE for both static and dynamic prediction, with lower values indicating better accuracy.
  • Scaling properties: OUTLETS’ advantage over the MLP baseline grows with more training data in the dynamic setting.The Transformer-based speculative backbone continues to benefit from additional supervision, unlike a simple MLP lacking long-range inductive bias.
  • Computational overhead: 0.7 ms: the prediction head’s runtime cost per decoding step, compared with 1.7 ms for the speculative decoding backbone.This low-overhead characterization assumes the speculative backbone is already available; its cost matters when OUTLETS is used standalone.
  • System-level evaluation: The system study evaluates OUTLETS as a control-plane prediction signal under saturated serving with one prefill instance and three decode instances.Speculative acceleration is disabled so the service model remains fixed and improvements can be attributed to routing and queue ordering.
  • System-level evaluation: 34.8%: short-request P99 latency reduction at 100 QPS, from 59.8 s to 39.0 s, while peak throughput rises from 17.8k to 18.4k tokens/s.The comparison replaces RR+FCFS with OUTLETS-guided LB+SJF; local SJF accounts for most tail-latency reduction, while load balancing adds throughput improvement.
  • Ablation studies: Ablations show that joint length-regression training does not materially reduce speculative acceptance, while removing the draft decoder causes the largest degradation.OUTLETS is generally comparable to or better than LP-ONLY for static prediction, although LP-ONLY has a modest dynamic-prediction advantage.

6 Conclusion

OUTLETS repurposes speculative-decoding backbones for output-length prediction. Across evaluated models and datasets, its speculative-drafter representations improve prediction accuracy, and its static predictions support load balancing and short-request prioritization in disaggregated serving.

  • Contribution: OUTLETS repurposes speculative-decoding backbones for output-length prediction across evaluated models and datasets.The framework uses representations produced by the speculative drafter rather than treating them only as token-drafting features.
  • Prediction: Lower prediction MAE: speculative-drafter representations outperform shallow MLP probes over target-model hidden states.When the representations are already computed for speculative decoding, OUTLETS adds only a lightweight regression head.

Limitations

OUTLETS has its lowest marginal deployment cost when a speculative backbone already exists, but standalone use must account for backbone cost. The system evaluation is limited to static admission-time scheduling, a fixed decoding setup, and outputs up to 2,048 tokens.

  • Standalone length prediction must include the speculative backbone's cost, so lighter predictors may be preferable in extremely latency-sensitive settings.
  • The system study uses static predictions for admission-time load balancing and SJF, not online migration or rescheduling.
  • Evaluation is bounded by 2,048-token outputs and a fixed decoding setup, leaving other decoding policies and very long trajectories unverified.

B Experimental Details

Experiments train and evaluate predictors across regenerated, length-filtered datasets and multiple model regimes. Baselines include neural regressors, classification, and prompting, while serving tests use a filtered ShareGPT workload in a disaggregated system.

  • Training and data: Training uses AdamW with learning rate 1e-5, weight decay 0.02, 2,000 warmup steps, gradient clipping at 1.0, BF16 precision, and 2,048-token truncation.
  • Training and data: Datasets comprise ShareGPT, Alpaca, and LMSYS-Chat-1M, with model outputs regenerated to obtain consistent ground-truth lengths and samples longer than 2,048 tokens removed.
  • Training and data: Qwen3-30B-A3B uses first-turn labels for multi-turn ShareGPT and LMSYS data and a 1/10 LMSYS subsample because of accounting and computational constraints.
  • Model regimes: The evaluation covers dense instruction-following models and the reasoning-capable Qwen3-30B-A3B MoE model, whose generations are longer and more variable.
  • Baselines: Baselines include BERT and OPT regressors, 50-bin classification, and PIA-style suffix prompting that asks for a length estimate before response generation.
  • Serving evaluation: Serving evaluation uses one prefill instance, three decode instances, and 1,914 ShareGPT requests with prompt lengths of at most 50 tokens.

C.1 Compared Configurations

The study compares a standard round-robin FCFS baseline with prediction-driven SJF and adds instance-level load balancing to separate local queue ordering from global routing. Speculative acceleration is disabled so serving differences reflect scheduling decisions.

  • RR + FCFS is the baseline, using round-robin dispatch and FCFS admission in a standard vLLM backend.
  • RR + SJF combines round-robin dispatch with local shortest-job-first scheduling driven by OUTLETS static length predictions.
  • LB + SJF routes requests to the least-loaded decode instance and applies local shortest-job-first scheduling.
  • All configurations disable speculative acceleration, keeping the decoding service model fixed while comparing routing and queue-ordering decisions.

C.2 Results Under Saturation

Under 100-QPS saturation, prediction-driven SJF reduces short-request tail latency, while load balancing additionally improves aggregate throughput. Benefits extend across request lengths, and the study finds no long-job starvation in the full configuration.

  • Throughput: 18,434.7 tokens/s: LB + SJF reaches the highest aggregate throughput versus 17,840.4 tokens/s for the baseline, a 3.3% improvement.
  • Long-task effects: 100.1s: LB + SJF lowers long-task P99 latency from the 103.0s baseline, indicating no observed starvation of long jobs.
  • Evaluation protocol: The serving comparison uses 1,914 requests averaging approximately 1,127 output tokens, with each complete experiment repeated five times.
  • Predictor comparison: More accurate predictors generally lower latency under both routing policies and usually improve throughput when paired with prediction-based load balancing.
  • Predictor comparison: Prediction accuracy and scheduling outcomes are not strictly monotonic because SJF depends on relative job ordering rather than pointwise length error.

D.1 Impact of Joint Training

OUTLETS jointly trains a speculative backbone for token drafting and remaining-length regression, testing whether both objectives can coexist without materially harming drafting quality. It remains competitive with LP-ONLY for static prediction, while accepting a modest dynamic-accuracy trade-off for lower deployment overhead.

  • Joint-training design: OUTLETS jointly trains one speculative backbone for autoregressive drafting and remaining-length regression.This design avoids introducing a separate auxiliary model while exposing shared representations to both objectives.
  • Joint-training design: The shared backbone preserves local drafting quality while its regression head extracts broader trajectory and termination signals.The authors describe the regression head as largely consuming information already present in the shared representation.
  • OUTLETS vs. SD-ONLY: OUTLETS and SD-ONLY have closely matching speculative acceptance rates across drafting steps, indicating no material acceptance degradation from the auxiliary objective.The comparison uses Figure 6; acceptance rate is the primary indicator of alignment between draft proposals and target-model verification.
  • OUTLETS vs. LP-ONLY: In static prediction, OUTLETS generally matches or outperforms LP-ONLY, whereas LP-ONLY consistently achieves lower MAE in dynamic prediction.The static setting provides one target per request, while dynamic prediction supplies a remaining-length target at every decoding step.
  • OUTLETS vs. LP-ONLY: OUTLETS trades a modest dynamic-prediction accuracy gap for lower memory overhead and tighter serving-pipeline integration than an additional model.This trade-off is especially relevant because static predictions support pre-decode dispatch and initial load balancing.

D.2 Feature Importance Analysis

Feature ablations identify the draft decoder as the key component for accurate length prediction, while direct regression is preferred to bucket classification. A cross-domain experiment provides preliminary evidence that OUTLETS transfers beyond chat-style workloads.

  • Feature Importance Analysis: Removing the draft decoder causes a substantial MAE increase, showing that fused hidden states alone are insufficient for length prediction.The ablation isolates the draft decoder among the main EAGLE-3-style backbone components.
  • Feature Importance Analysis: Adding the length-prediction objective does not introduce an additional speculative-acceptance drop as the drafting horizon increases.Acceptance rate typically decreases with horizon because mismatches compound, while EAGLE-3's TTT mechanism already alleviates that degradation.
  • Feature Importance Analysis: The draft decoder transforms fused features into trajectory-aware representations that capture progress toward generation termination.These findings support reusing the speculative backbone as a forecasting module, with the draft decoder as its key component.
  • Prediction formulation: Direct regression consistently outperforms bucket-based classification because it preserves numerical distance between predicted and target lengths.Regression also supplies the scalar estimate typically required by downstream scheduling.
  • Cross-domain evaluation: On GSM8K and HumanEval, MAEs are 25.5% and 21.8% of the corresponding average output lengths after training only on LMSYS.The experiment increases maximum sequence length to 4,096 tokens and evaluates OUTLETS without mathematical-reasoning or code-generation training data.
Loading 2609.01068v1…