PaperlayerView exact revision on arXiv

Source-linked AI summary

vSkipper: Translating Dynamic Layer Skipping into LLM Serving Gains

Wei Da, Yavuz Ferhatosmanoglu, Evangelia Kalyvianaki

arXiv:2609.37062v1cs.DCcs.LG

TL;DR

Dynamic layer skipping reduces computation, but existing serving engines assume regular fixed-depth execution, so saved work does not necessarily yield serving gains. vSkipper makes per-token skipping pluggable while preserving engine interfaces; against upstream SGLang it reduces latency at the load-curve knee and increases throughput under saturation.

  • Problem

    Dynamic skipping reduces per-token computation, but modern serving engines assume regular batches whose tokens traverse every layer, leaving an interface gap between skippers and engine batching, cache, and graph management.

    Source: §1 · p. 1

  • Method

    vSkipper provides a reusable serving virtualization layer that groups tokens by skip decision into RUN and Project-Only cohorts while preserving engine batching, cache, and graph interfaces.

    Source: §1 · pp. 1–2, §3.1 · p. 4

  • Results

    36.8% lower mean end-to-end latency on GSM8K and 13.6% on BBH at upstream’s load-curve knee; saturated request throughput rises by 11.3% and 7.4%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The same implementation extends across synthetic skip policies, two Qwen3 skippers, workloads, and three GPUs without workload-specific tuning.

    Source: §6 · p. 10

Abstract

from arXiv · show

Dynamic layer skipping reduces LLM computation by allowing each token to execute only a subset of the model's layers. However, existing skippers rely on specialized generation loops and do not integrate with modern serving engines. As a result, fewer executed layers do not necessarily translate into lower serving latency: FlexiDepth skips 8 of Llama-3-8B's 32 layers on average, yet its standard generation loop decodes 14.6--21.0% more slowly than the base model. We present vSkipper, a virtualization layer that makes dynamic layer skippers pluggable in serving engines while preserving continuous batching, fixed-shape batches, paged KV caching, and captured decode graphs. At each routed layer, vSkipper groups tokens by the skipper's decision and uses routed execution only when predicted to be profitable. We implement vSkipper in SGLang and evaluate the released FlexiDepth checkpoint against upstream SGLang under identical prompts, arrivals, output lengths, and launch settings. At the knee of upstream's load curve, vSkipper reduces mean end-to-end latency by 36.8% on GSM8K and 13.6% on BBH. Under saturation, it increases request throughput by 11.3% and 7.4%. Serving adds no statistically resolved quality loss beyond the checkpoint's own. Across synthetic skip policies, two Qwen3 skippers, and three GPUs, we demonstrate reuse without workload-specific tuning. To our knowledge, vSkipper is the first system to realize serving-efficiency gains from per-token interior layer skipping within a modern LLM serving engine. The code is open-sourced as an SGLang fork at https://github.com/AKafakA/sglang-vskipper/tree/vskipper-ref

1 Introduction

Dynamic layer skipping can reduce computation, but integrating it with batched serving requires an execution interface that preserves the engine’s batching, cache, and graph interfaces. vSkipper supplies that interface and reports lower latency and higher saturated throughput against upstream SGLang.

1 Introduction

  • Dynamic layer skippers choose which layers each token executes, but translating saved computation into batched serving gains requires a separate execution solution.Modern engines rely on regular batching, paged KV caching, and captured GPU graphs that assume tokens traverse every layer.Source: §1 · p. 1
  • vSkipper makes per-token interior layer skipping a pluggable serving-engine component while preserving batching, cache, and graph interfaces.Its RUN/Project-Only interface supports mixed-depth execution within captured graphs, with profitability-aware mode switching.Source: §1 · pp. 1–2
  • At upstream’s load-curve knee, vSkipper lowers mean end-to-end latency by 36.8% on GSM8K and 13.6% on BBH; under saturation, throughput rises by 11.3% and 7.4%.Serving adds no statistically resolved quality loss beyond the checkpoint’s own.Source: §1 · p. 2
  • The controlled evaluation compares vSkipper with upstream SGLang under matched launch settings, prompts, arrivals, and output lengths.The implementation also generalizes across nine synthetic policies, two Qwen3 skippers, three workloads, and three GPUs without workload-specific tuning.Source: §1 · p. 2

2 Background and Motivation

Modern serving engines assume fixed-depth batch execution, while dynamic skipping gives tokens different layer paths. A standard generation loop can therefore lose throughput despite skipped computation, motivating an abstraction that preserves batch efficiency and correct per-layer state.

2 Background and Motivation

  • Despite skipping 8 of 32 layers on average, FlexiDepth decode throughput falls by 21.0% at batch size 1 and 14.6% at batch size 8.The comparison uses the public checkpoint over 100 prompts with 256 greedy output tokens.Source: §2 · p. 2
  • vSkipper’s overview combines profitability-based dense/routed mode selection with routed-layer cohorts whose Project-Only rows still write their own-layer KV.Rows are grouped by skipper decision and scattered back to the batch.Source: Figure 2 · p. 3
  • At batch size 1, routing overhead offsets saved work; at batch size 8, the batch traverses the union of tokens’ required layers, while splitting sacrifices batch width.The standard loop’s lower throughput is reported for the FlexiDepth checkpoint against its base model.Source: §2 · pp. 2–3
  • A serving abstraction must preserve batch efficiency, maintain correct per-layer state across divergent paths, and route only when savings exceed overhead.Even a row skipping a layer’s main computation must produce the KV state future tokens need.Source: §2 · p. 3

3 Design and Implementation

vSkipper separates the skip policy from serving-engine state, executing mixed-depth token paths while maintaining cache correctness. It uses profitability-aware mode switching and implements the interface in an SGLang fork.

3 Design and Implementation

  • vSkipper separates skipping policy from execution and engine state, converting per-token decisions into route data, packed cohorts, and KV-complete cache state.The engine retains continuous batching, admission control, paged KV management, and prefix reuse.Source: §3 · p. 3

3.1 Route-Guided Cohort Execution

  • Routed layers record decisions on a route tape, execute RUN and Project-Only cohorts separately, then scatter updates; Project-Only rows still write their own-layer KV.The device-resident tape allows routes to change without host synchronization or graph recapture; fixed-grid GEMMs serve varying cohort sizes in captured decode graphs.Source: §3.1 · pp. 3–4

3.2 Invariants for Correct Mixed-Depth Execution

  • Correct mixed-depth execution requires every layer to write its own KV and every route-dependent row-to-memory map to be recomputed when the row set changes.These invariants protect paged KV state and prevent stale maps from directing requests to other requests’ KV pages.Source: §3.2 · p. 5

3.3 Profitability-Aware Mode Switching

  • The mode switch selects dense or routed execution separately for prefill and decode, using routing only when estimated savings cover its fixed cost.Decode promotion can occur once after prefill; prefill routing requires at least 1,536 prompt tokens and an estimated Project-Only share above 0.35.Source: §3.3 · p. 5

3.4 Implementation

  • The SGLang fork leaves its multiprocess architecture, admission control, and memory management unchanged, and supports Llama-3 and Qwen3 routed-layer hooks.Policies include trained gates and a seeded hash; a new policy can reuse an existing projector and execution path.Source: §3.4 · p. 5

4 Evaluation

The evaluation tests whether vSkipper improves serving at matched work without adding resolved quality loss, and whether its execution framework generalizes across policies, models, and hardware.

4.1 Experimental Setup

  • Evaluation used GSM8K, BBH, and CoQA with seeded Poisson arrivals, matched per-request output lengths, and identical launch settings.Both arms replayed the same prompts and arrival schedule; saturated throughput was measured within the injection window.Source: §4.1 · pp. 6–7

4.2 Serving Comparison (Q1)

  • At matched work, mean E2E latency fell 36.8% on GSM8K and 13.6% on BBH at their knees; under saturation, RPS rose 11.3% and 7.4%, respectively.The latency comparisons are against upstream SGLang; saturation is measured at 1.25 × Q∗.Source: Table 2 · p. 7, §4.2 · pp. 7–8, Table 4 · p. 8
  • GSM8K’s knee gain came from prefill routing alone (−35.6%), whereas BBH’s came from decode routing (−16.0%); always-route was +5.5% slower than upstream at the CoQA knee.Disabling promotion reduced the GSM8K knee gain to −32.6%.Source: §4.2 · pp. 7–8, Table 3 · p. 8
  • For CoQA, 2.0% of generations ran past 4,096 tokens and carried 94% of a cell’s output tokens; vSkipper improved makespan by 7.0–7.8% at every rate.The gain was concentrated in long requests.Source: §4.2 · p. 8

4.3 Task Quality (Q2)

  • Serving added no resolved quality loss beyond the checkpoint’s own: DiD intervals included zero on GSM8K, BBH, and CoQA, and the hybrid showed no resolved knee difference from always-route.The reported DiD values were −0.23 ± 0.68 pp, +0.45 ± 0.55 pp, and +0.07 ± 0.12 pp, respectively.Source: §4.3 · p. 8, Table 5 · p. 8

4.4 Generality (Q3)

  • Across nine RandomSkip arms, always-route cost up to +23.8%; switching reduced unprofitable-policy costs, while gains reached −49.1% at r=d=75%.The figure compares routed-row share r and skipped-layer share d under three switch settings.Source: §4.4 · p. 9, Figure 3 · p. 9
  • At the GSM8K knee, Qwen3-4B showed no resolved mean E2E change (+2.7 ± 2.8%), while Qwen3-8B lowered mean E2E by 6.6 ± 0.3% without resolved added quality loss.The Qwen3-8B result is for a checkpoint with an always-route decode skip of 0.416.Source: §4.4 · p. 9, Table 6 · p. 9
  • On GSM8K at each device’s knee, mean E2E fell by 39.4 ± 0.2% on the H100 and 30.0 ± 4.2% on the RTX A6000.The same implementation used rule-derived thresholds without workload-specific tuning.Source: §4.4 · p. 10, Table 7 · p. 10

5 Related Work

Prior work addresses which layers or sublayers to skip, but implementations commonly use specialized execution paths. vSkipper instead targets integration of per-token layer skipping with modern serving engines.

5 Related Work

  • Dynamic-skipping methods select layers per token, skip sublayers, use fixed or scheduled subsets, exit early, or remove layers.Other methods constrain the policy to simplify execution.Source: §5 · p. 10
  • Released FlexiDepth, AdaSkip, and LoRA-Drop implementations use specialized PyTorch paths, whereas vSkipper provides a RUN/Project-Only interface for modern serving engines.Source: §5 · p. 10
  • Early-exit systems remove suffixes of layers, adapter-serving systems retain fixed layer traversal, and MoE routes across experts within a layer; vSkipper routes across depth while preserving per-layer KV state.Source: §5 · p. 10

6 Conclusion and Future Work

vSkipper combines a pluggable per-token skipping interface with mixed-depth execution and profitability-aware switching, while retaining serving-engine interfaces. The conclusion reports gains without statistically resolved serving-induced quality loss and identifies broader execution support as future work.

6 Conclusion and Future Work

  • vSkipper converts FlexiDepth’s saved computation into lower latency and higher saturated throughput without statistically resolved serving-induced quality loss, and generalizes across policies, Qwen3 skippers, workloads, and GPUs without workload-specific tuning.The authors identify support for additional engines and attention backends, sublayer skippers, model families, and distributed serving as future directions.Source: §6 · p. 10

A A Minimal Skipper Plugin

  • The example static-depth plugin selects a tail fraction of routed layers for Project-Only execution and registers its RUN and Project-Only actions with the serving configuration.At ratio zero every routed layer runs; at ratio one every valid row projects throughout the routed set.Source: A A Minimal Skipper Plugin · pp. 13–14

B Route-Guided Cohort Execution at the Kernel Level

  • Route-guided cohort execution packs rows by decision, executes their distinct paths, and scatters outputs back while preserving the KV state future attention requires.Prefill uses cuBLAS over packed rows; captured decode uses count-bounded Triton GEMMs.Source: B Route-Guided Cohort Execution at the… · p. 14

C Validation Gates

  • Serving gates verify served configuration, cross-arm comparability, valid work accounting, and evidence that the declared skipping mechanism actually ran.The checks can refuse a cell when a required identity, work, or mechanism condition fails.Source: C Validation Gates · p. 14

D The Switching Thresholds from the Roofline

  • The switching rule estimates when routed execution is profitable from resident tokens, skip ratio, routed layers, context traffic, and bandwidth, then sets device and model thresholds without per-device retuning.The A100 rule’s predicted exit threshold is within 2% of the measured exit threshold.Source: D The Switching Thresholds from the… · p. 15

E.1 First-Token Agreement

  • On the two prompts with empty generations, each served arm matches its native PyTorch reference within 0.01 nats, and the checkpoint selects an immediate stop.The checkpoint’s first-token choice is <|eot_id|> on both prompts.Source: §E.1 · p. 16, Table 9 · p. 16

E.2 GSM8K Answer Extraction

  • A composite GSM8K extraction rule uses strict-match when it parses and otherwise flexible extraction, preserving the base-model scores while correctly reading checkpoint confidence epilogues.Its checkpoint cost is −3.26 pp, between the two single-filter values; the reported single-filter DiD intervals agree with the composite result’s interval.Source: §E.2 · pp. 16–17

E.3 The Served Configuration Across Loads

  • Across the load grid, hybrid routing shows no resolved quality difference from upstream below the knee and none from always-route at or above it; BBH routes 99.0% of decode rows at the knee.Table 10 covers served-system quality at each rate; the knee comparison is its Table 5 right-hand row.Source: §E.3 · p. 17, Table 10 · p. 17

F Latency and Makespan Across Load

The load analysis interprets latency and makespan under open-loop arrivals, then distinguishes matched-work runtime comparisons from natural-stopping stack behavior. It reports workload-specific throughput and output-length patterns.

F Latency and Makespan Across Load

  • The open-loop makespan includes the injection window and final drain, so execution savings appear in latency while the server keeps up and makespan reflects the longest-request tail.Figure 5 plots mean E2E and makespan changes against offered load per workload.Source: F Latency and Makespan Across Load · p. 17, Figure 5 · p. 18
  • The mean E2E reduction is roughly uniform across output lengths at and below the knee, but its per-bin spread widens to 8.0 pp on overloaded GSM8K and 7.8–11.1 pp on CoQA.On CoQA, a request’s place in the queue sets its gain.Source: F Latency and Makespan Across Load · p. 18
  • On BBH under saturation, request throughput rises by 7.4% while token throughput rises by 4.1%, as in-flight requests grow during the shared injection window.On GSM8K and CoQA, the running batch stays full while the waiting queue grows, and TPS and RPS changes agree within one percentage point.Source: F Latency and Makespan Across Load · p. 18

G Natural Stopping and the Output Lengths

  • Under natural stopping at the GSM8K knee, the checkpoint generates 330 output tokens per request versus the base model’s 116 (+185%) and hits the context limit on 99 requests versus 3; these are stack-level differences, not runtime effects.On BBH it generates 53–59% more tokens; on CoQA it generates similar or less work (−3 to −24% output tokens) and its stack finishes sooner (E2E −4 to −26%).Source: G Natural Stopping and the Output… · p. 18, Table 11 · p. 19

G.1 Native Checkpoint Repetition Analysis

The native repetition analysis compares FlexiDepth with its base model on the same prompts and reports prompt-specific repetition behavior. The evaluation keeps greedy decoding and leaves sampling-penalty tuning for future work.

G.1 Native Checkpoint Repetition Analysis

  • On a seeded sample of 500 GSM8K knee-cell prompts, the checkpoint repeats on 2.2 ± 1.3% while the base model repeats on none, with each repetition a degenerate answer-line repeat.Both generate greedily from frozen token ids within an 8192-token window.Source: §G.1 · p. 19
  • Repetition is prompt-specific: 20.3% of the 143 served-arm repeaters repeat in the checkpoint, versus 2.0% of controls.The sampled groups stop once repetition is detected.Source: §G.1 · p. 19
  • The evaluation uses greedy decoding under the third-party lm-eval protocol and leaves sampling-penalty tuning as future work.Which prompts repeat under serving varies with load and stack.Source: §G.1 · p. 19

H Mechanism Ablation

The ablation separates phase routing and switching effects, while kernel comparisons and tail-latency reporting characterize execution costs and additional outcome measures. The absolute-baseline appendix documents workload and serving conditions.

H Mechanism Ablation

  • Prefill routing supplies the GSM8K knee-cell E2E gain (−35.6%, near the served arm’s −36.8%); BBH gains from decode routing, and the switch avoids routed-mode costs on passes kept dense.On CoQA, served and decode-only arms improve makespan by 7.0–8.1%; at the knee, always-route trails the served arm by 9.5 pp in mean E2E.Source: Table 13 · p. 20, H Mechanism Ablation · p. 20

H.1 Kernel Headroom

  • The served count-bounded kernels take a median 1.18× cuBLAS time on A100 and 1.13× on RTX A6000, and the reported serving gains already include these kernels.The comparisons identify target shapes for further kernel optimization.Source: §H.1 · p. 21

I Tail Latencies

  • Figure 6 compares p99 E2E change against upstream across nine RandomSkip arms and three switch settings; Table 14 reports p50 and p99 changes with 95% intervals.The supplied passages specify the scope of these tail-latency results, not their values.Source: Figure 6 · p. 21, Table 14 · p. 21, I Tail Latencies · p. 21

J Absolute Baseline Numbers

  • The baseline appendix reports per-cell upstream absolutes and notes that CoQA’s drain is 61–76% of makespan at every load; both arms enable the radix prefix cache with cached prefill-token shares within 0.8 pp.Both arms capture decode graphs up to 1,024 rows; vSkipper keeps separate dense and routed prefix-cache namespaces keyed by prefill mode.Source: J Absolute Baseline Numbers · pp. 21–22, Table 15 · p. 22

K Two Qwen3 Skippers: the Profitability Boundary

The Qwen3 comparison tests when dynamic skipping becomes profitable and how skip rate relates to serving gains. The highest-skipping Qwen3-8B checkpoint shows the largest latency reductions, while Qwen3-4B shows no resolved mean end-to-end change.

Profitability setup

  • The two alignment-only straight-through-gate Qwen3 routers and projectors target layers 18–35 of 36, with thresholds calibrated from 32-request probes estimating skip shares of 0.25 for 4B and 0.445, 0.485, and 0.520 for the three 8B checkpoints.The matched-work protocol uses each arm’s own upstream Q* and output lengths.Source: K Two Qwen3 Skippers: the… · p. 22

Serving

  • The highest-skipping Qwen3-8B checkpoint lowers mean E2E latency most at every load, with intervals excluding zero; Qwen3-4B shows no resolved mean E2E change at any load.Table 17 reports matched-work percentage changes and 95% intervals for the Qwen3 checkpoints against upstream.Source: Table 17 · p. 23, K Two Qwen3 Skippers: the… · p. 23

Quality and tuning

  • Along the 2×10^-4 run, later checkpoints skip more and score higher natively and when served; all four difference-in-differences intervals contain zero at one run per arm.The checkpoint comparison reports no resolved loss beyond each checkpoint’s own in the routed comparison.Source: K Two Qwen3 Skippers: the… · p. 23
  • These Qwen3 skippers are proof-of-concept alignment-only runs, not evidence of the best quality–efficiency trade-off attainable with full two-stage training.The authors leave further hyperparameter tuning of the skipper’s quality–efficiency trade-off to future work.Source: K Two Qwen3 Skippers: the… · p. 23

Serving

  • The separately measured hybrids route 86.5% of 4B and 99.3% of 8B decode rows at the knee, scoring +1.52 ± 2.49 and +4.25 ± 2.68 pp against always-route, respectively.Source: K Two Qwen3 Skippers: the… · p. 23

L Hardware Transfer

The same implementation transfers to H100 and RTX A6000 using device-specific load grids and decode settings. Latency reductions persist without workload-dependent configuration search, with an A6000-specific tile choice required by its shared-memory limit.

Latency transfer

  • On H100, end-to-end latency falls on every row, most at the knee (−39.4%); on RTX A6000, mean latency falls on every row, with intervals excluding zero at the knee and in overload.Each device is measured on its own load grid; H100 has Q*=14 req/s and RTX A6000 has Q*=6 req/s.Source: L Hardware Transfer · p. 24

Protocol and thresholds

  • Each device uses its own knee, output lengths, decode-graph range, and three paired repetitions; thresholds follow the bandwidth-based rule, with no workload-dependent configuration search.The repeated protocol uses the same code, launch profile, suite, gates, and prefill floor.Source: L Hardware Transfer · pp. 23–24, Table 19 · p. 24

Device-specific implementation

  • On RTX A6000, the A100 cohort tiles exceed the card’s 101 KB shared-memory limit, so the tuner replaces 19 of 20 tile keys while preserving bit-identical outputs.The A6000 decode graphs capture up to 256 rows, compared with upstream’s default limit of 32 on that card.Source: L Hardware Transfer · p. 24

M Reproducibility

The reproducibility appendix identifies the evaluated model and skipper checkpoints, Qwen3 training recipe, serving-engine baseline, and rerun costs. It also records the checks and configuration controls used to keep the comparisons auditable.

Models and checkpoints

  • The Llama evaluation uses Meta-Llama-3-8B-Instruct and the released FlexiDepth checkpoint, which routes the last 16 of 32 layers; Qwen3 checkpoint revisions and exports are also specified.The Qwen3 base models are served with thinking disabled through the chat template.Source: M Reproducibility · p. 25

Qwen3 training

  • The Qwen3 skippers are trained over frozen bases with straight-through hard gates on Tulu-3 data, using sequence length 2,048, global batch 32, and at most one epoch.Checkpoint selection probes, router-penalty schedules, and training GPU allocations are specified.Source: M Reproducibility · p. 25

Engine and configuration

  • The baseline is upstream SGLang at commit 602c8615a1, and vSkipper adds the runtime to that commit; shared launch settings, graph captures, boot checks, and per-cell switch constants are documented.Source: M Reproducibility · p. 25

Rerun cost

  • Measured serving cells total about 34 A100 GPU-hours, with H100 and RTX A6000 cells taking 1.8 and 4.2 GPU-hours; reruns of the serving evidence cost under USD 100 at the listed rental prices.Rate sweeps, quality cells, length-recording runs, and restarts roughly double the A100 total.Source: M Reproducibility · p. 25

Related papers

Loading 2609.37062v1…