A rigor-matched audit of periodic-step layer skipping for efficient llm inference: conflayers versus swift, with a supplemental analysis of trained routing alternatives

Prateek Kumar Sikdar; Arpan Ghosh

Source-linked AI summary

A rigor-matched audit of periodic-step layer skipping for efficient llm inference: conflayers versus swift, with a supplemental analysis of trained routing alternatives

Prateek Kumar Sikdar, Arpan Ghosh

arXiv:2608.28846v1Aug 28, 2026cs.CLcs.AIcs.LG

TL;DR

Efficient LLM layer skipping lacks rigor-matched comparisons across methods with different decision granularities. The paper audits periodic-step ConfLayers and SWIFT under matched conditions, finding SWIFT stronger on accuracy and pure-inference speed, while coarser trained-routing methods show modest speedups but weaker accuracy.

Problem
Layer-skipping methods make decisions at different times and input granularities, limiting direct comparisons of their accuracy and efficiency.
Method
The paper conducts a three-seed audit of ConfLayers and SWIFT against vanilla decoding across two model scales and two tasks, with separated search-cost accounting and supplemental verified evaluations of LayerRoute and LayerDrop.
Results
SWIFT is stronger than ConfLayers on accuracy and decomposed inference speed, while LayerRoute and LayerDrop deliver modest genuine speedups but substantially lower accuracy.
Takeaways & Limitations
Rigor-matched comparisons should align decision granularity and report online-search overhead separately from pure inference cost.
Takeaways & Limitations
The audit covers two instruction-tuned model scales and two tasks, uses 100-sample evaluations and three seeds, and omits several deployment factors.

Abstract

from arXiv · show

Layer-skipping methods for efficient LLM inference decide, at some granularity, which transformer layers to execute for a given input. We present a rigor-matched, three-seed audit of two periodic-step, search-based methods that make this decision online at inference time and re-evaluate it every few generation steps: a confidence-gated early-exit baseline (ConfLayers) and genuine self-speculative decoding (SWIFT, Xia et al. 2024), together with vanilla autoregressive decoding, across two model scales (Qwen2.5-0.5B and Qwen2.5-1.5B) and two tasks (GSM8K reasoning and CNN/DailyMail summarization). SWIFT is the strongest method on accuracy in three of four cells; ConfLayers is dominated everywhere, with particularly large deficits on GSM8K at 1.5B. Once online-search overhead is separated from pure inference cost, SWIFT's true inference speed is faster than ConfLayers's in all four cells (5-21%), reversing the naive wall-clock ranking in three of them. ConfLayers's search overhead is small and stable (1-2% of cost), while SWIFT's is larger and more variable (up to 28.7%). We additionally examine two trained-routing methods, LayerRoute (Sikdar, 2026) and LayerDrop (Fan et al. 2020), as a supplemental analysis because they operate at coarser decision granularities. Under a verified protocol with genuine per-input gating, a genuine full-model baseline, and genuine inference-time compute skipping, both show modest speedups (1.08-1.33x) but accuracy well below the periodic-step methods, including a near-total collapse for LayerRoute on GSM8K at 1.5B (0.003 mean exact-match across three seeds). We release the full audit protocol as a template for rigor-matched efficiency comparisons.

1 INTRODUCTION

The paper argues that fair layer-skipping comparisons must match decision granularity, and audits ConfLayers and SWIFT under shared periodic-step, search-based conditions.

Layer-skipping methods differ in when and at what input granularity they decide whether to execute or bypass transformer layers.Confidence-gated methods threshold per-token confidence, whereas self-speculative methods draft with a cheapened model and verify against the full model.
The main comparison matches ConfLayers and SWIFT on periodic-step online search, seeds, tasks, scales, and cost accounting.The audit separates online search overhead from the inference cost that search accelerates.
Excluding search overhead reverses SWIFT and ConfLayers’s pure-inference speed ranking in three of four tested task/scale combinations.This demonstrates why naive wall-clock comparisons can misrepresent efficiency when methods incur different search costs.
LayerRoute and LayerDrop are evaluated separately because their decision granularities do not match the periodic-step methods.The supplemental protocol verifies genuine per-input gating, a genuine full-model baseline, and genuine inference-time compute skipping.
The paper releases a reusable rigor-matched audit protocol for efficiency comparisons.The protocol includes a frozen-replay search-overhead decomposition and explicitly distinguishes methods by routing granularity.

2 A TAXONOMY OF LAYER-SKIPPING DECISION GRANULARITY

The taxonomy classifies layer-skipping methods by how often and for which input unit they make skip decisions, establishing which methods are directly comparable.

Table 1 organizes methods by the frequency and input unit of their skip decisions.This dimension is presented as the basis for comparability.
ConfLayers and SWIFT periodically re-search configurations every 25–30 generation steps, forming the matched comparison regime.Their found configuration may converge early and then remain frozen for the rest of a session.
LayerRoute makes one input-conditioned gate decision per sequence, a coarser regime than periodic-step routing.During genuine generation, its decision can drift toward continuously updating behavior absent an explicit freezing mechanism.
LayerDrop applies one fixed pruning pattern to every input, making it the coarsest and least adaptive regime.Its pattern is chosen at training time and does not vary with the input.
Comparing the periodic-step methods directly against LayerRoute or LayerDrop would compare methods answering different questions.The trained-routing methods are therefore reported as a supplemental analysis rather than head-to-head peers.

3 RELATED WORK

Related work spans trained routing, confidence-gated early exit, speculative decoding, and structural pruning, with methods differing substantially in routing granularity and mechanism.

Trained routing and structured skipping: LayerDrop uses structured dropout to train transformers to tolerate layer removal, enabling inference-time depth reduction without retraining.Other trained-routing approaches learn per-example halting or route a fixed fraction of tokens through each layer.
Confidence-gated and search-based early exit: ConfLayers extends confidence-gated early exit with online Bayesian optimization over per-layer thresholds.Its confidence-gated lineage includes CALM and BranchyNet-style early-exit methods.
Speculative and self-speculative decoding: SWIFT performs self-speculative decoding by self-drafting with a layer-skipped version of the target model instead of using a separate draft model.Speculative decoding generally drafts multiple tokens with a cheap model and verifies them in one batched pass.
Compute-efficient scaling and pruning: The paper relates scale-dependent behavior in layer skipping to compute-optimal scaling and structural pruning methods that find removable redundancy depends on scale.The cited pruning methods include LLM-Pruner and Wanda.

4 BACKGROUND: METHOD FORMALIZATIONS

The formalizations define ConfLayers as confidence-threshold early exit with online threshold search and SWIFT as lossless self-speculative decoding with online skip-set search.

4 BACKGROUND: METHOD FORMALIZATIONS: ConfLayers and SWIFT are formalized here, while LayerRoute and LayerDrop are formalized in the supplemental analysis.
4.1 THE CONFLAYERS BASELINE: ConfLayers wraps a fixed pretrained backbone with per-layer confidence rules and Bayesian optimization over thresholds, requiring no training.Threshold candidates are evaluated on a held-out probe set and used to update a surrogate cost–quality model.
4.1 THE CONFLAYERS BASELINE: At layer l, ConfLayers projects the hidden state through a shared frozen language-model head and uses the top vocabulary probability as confidence.The confidence signal is compared with the layer-specific threshold τ_l.
4.1 THE CONFLAYERS BASELINE: ConfLayers exits at the first layer clearing its threshold, emits that layer’s logits, and skips all subsequent layers.The operational exit layer is l* = min{l : z_l = 1}.
4.2 THE SWIFT BASELINE: SWIFT constructs a no-training self-draft model by replacing selected target-model layers with identity pass-throughs while reusing the original weights.The skip set S determines which layers are bypassed.
4.2 THE SWIFT BASELINE: SWIFT drafts k tokens autoregressively with the self-draft model and verifies all k positions in one batched pass of the full model.Rejected tokens are corrected using the standard rejection rule, preserving the full model’s output distribution exactly.
4.2 THE SWIFT BASELINE: SWIFT searches skip sets online, trading cheaper drafts against lower acceptance rates through Bayesian optimization over discrete skip sets.The search is separated from pure drafting cost in the paper’s cost decomposition.

5 EXPERIMENTAL SETUP

The audit evaluates vanilla decoding, ConfLayers, and SWIFT across two Qwen2.5 scales and two tasks using matched seeds, task sampling, and cost measurement procedures.

Models: Qwen2.5-0.5B and Qwen2.5-1.5B serve as the model backbones.
Tasks: GSM8K measures exact-match accuracy for multi-step reasoning, while CNN/DailyMail measures ROUGE-L for long-form summarization.Each task uses 100 held-out samples with an identical shuffle/seed recipe across dataset aliases.
Baselines: Vanilla decoding is the standard autoregressive baseline; ConfLayers and SWIFT are the evaluated acceleration methods.SWIFT results use a from-scratch Qwen2 port of the official self-drafting and tree-attention implementation, verified class-by-class against the reference.
Rigor protocol: Every method × task × scale accuracy and cost cell uses three seeds and reports the mean with the observed range.The protocol also verifies that CNN/DailyMail generation-length normalization does not itself explain score differences.
ConfLayers search calibration: ConfLayers thresholds are calibrated online per query using its reference implementation’s default search configuration.The supplied setup specifies a 30-query search interval, up to 100 optimization iterations, a 0.95 threshold cap, and a 100-token context window.
SWIFT search calibration: SWIFT’s online skip-set search uses reference defaults with a 50-token context window, Bayesian optimization every 25 steps, and up to 1,000 optimization iterations.Its upper-confidence-bound acquisition function uses κ=2.5, with a 0.93 maximum score threshold and 300 tolerance iterations.
Cost measurement: Latency is measured at batch size 1 on one GPU with explicit synchronization, including generation, sampling, KV-cache updates, and interleaved search.Tokenization and detokenization are excluded, and first-query initialization overhead is unsubtracted because no warm-up iterations were run.

6 RESULTS

Across the full evaluation matrix, ConfLayers is consistently weakest on accuracy, while decomposed cost accounting shows SWIFT’s pure inference is faster despite higher search overhead.

6.1 ACCURACY: The accuracy ranking is stable across all three seeds in every cell of the task × scale × method matrix.Figure 1 and Table 2 summarize the matrix for vanilla decoding and the two periodic-step methods.
6.1 ACCURACY: 0.077 versus vanilla’s 0.413 is ConfLayers’s GSM8K accuracy at 1.5B, its particularly large deficit in the evaluated conditions.ConfLayers underperforms both other methods in every condition.
6.1 ACCURACY: SWIFT is strongest in three of four cells, losing only to vanilla on GSM8K at 1.5B, where it scores 0.307 versus 0.413.SWIFT still substantially outperforms ConfLayers in that cell, 0.307 versus 0.077.
6.2 COST: 5–21% faster pure inference makes SWIFT faster than ConfLayers in all four cells after separating online-search cost.This reverses the naive wall-clock ranking in three cells.
6.2.1 SEARCH-OVERHEAD DECOMPOSITION: 1.2–2.2% of total cost is ConfLayers’s search overhead, compared with SWIFT’s 8.7–18.6% mean overhead.SWIFT’s overhead reaches 28.7% on individual GSM8K/0.5B seeds, whereas ConfLayers’s variance is near zero.
6.2.1 SEARCH-OVERHEAD DECOMPOSITION: Search state persists across queries, front-loading overhead onto early queries as both methods’ searches converge.The paper recommends reporting decomposition alongside total cost for N queries when N is known.

7 SUPPLEMENTAL ANALYSIS: TRAINED ROUTING AT COARSER GRANULARITY

The supplemental analysis evaluates LayerRoute and LayerDrop under a verified protocol with genuine gating, full-model baselines, and inference-time compute skipping. Both achieve modest speedups, but LayerRoute exhibits near-total GSM8K failure at 1.5B, while comparisons remain bounded by their coarser decision grananularity and specific training setup.

Scope and protocol: LayerRoute and LayerDrop are evaluated as supplemental methods because their coarser routing granularity does not match the periodic-step decisions of ConfLayers and SWIFT.LayerRoute makes per-sequence gate decisions, whereas LayerDrop applies one fixed pruning pattern to every input.
Method formalizations: LayerRoute attaches a scalar gate head and low-rank adapter at each transformer layer, producing a hard binary execution decision from the current sequence’s hidden states.The gate is trained jointly with LoRA adapters, using straight-through estimation for the binary decision.
Method formalizations: LayerDrop trains robustness to stochastic layer removal, then applies a fixed training-selected layer pattern at inference without a router or per-input adaptivity.Its fixed pattern makes LayerDrop the coarsest routing regime among the evaluated methods.
Measurement protocol: The measurement protocol verifies genuine per-input gating, a genuine full-model baseline, and genuine inference-time compute skipping rather than merely masking computation.LayerRoute’s gated pass differs from an ungated pass, the baseline executes all layers, and inference reduces real layer invocations.
Accuracy and cost: 0.003 mean exact-match is LayerRoute’s GSM8K accuracy at 1.5B, while both trained-routing methods deliver only 1.08–1.33× speedups over the genuine full-model baseline.LayerDrop’s speedup is 1.25–1.30× and is somewhat more consistent, despite lower average accuracy.
Interpretation and limitations: The results do not establish that trained routing generally underperforms search-based routing because only two methods, one fixed LoRA budget, and different decision granularities were examined.The authors characterize these methods as specific instances within a larger design space.

8 GENERALIZATION LIMITATION: LAYERROUTE ON TINYLLAMA

LayerRoute’s gating mechanism transfers to TinyLlama, but the skip-rate differential reverses direction despite nearly identical relative-depth skip patterns. The authors therefore identify architecture dependence as a limitation of generalizing the differential’s direction.

Generalization result: 3.766 < 4.705 and 6.133 < 7.784: gated perplexity is lower than full-computation perplexity for both tool-call and planning steps.This supports transfer of the gating mechanism’s quality behavior to TinyLlama.
Generalization result: 12.91% versus −7.92% to −3.11%: the skip-rate differential reverses between Qwen2.5-0.5B and TinyLlama-1.1B.Qwen2.5 skips more on tool-call steps, whereas TinyLlama skips more on planning steps.
Structural comparison: 32%–73% versus 33%–71% relative depth: TinyLlama and Qwen2.5 converge to nearly identical closed middle-band skip patterns.The comparison covers layers 7–15 of 22 for TinyLlama and 8–16 of 24 for Qwen2.5-0.5B.
Interpretation and limitation: The authors hypothesize that boundary layers near the threshold determine the differential’s sign, while the middle band is largely architecture-invariant.They explicitly describe this as a testable hypothesis rather than a confirmed mechanism.
Interpretation and limitation: The “tool calls skip more” claim does not universalize across model families, so deployments relying on its direction should re-verify it on the target architecture.The gating mechanism transfers, but the differential’s direction does not.

9 DISCUSSION AND LIMITATIONS

The audit finds SWIFT superior to ConfLayers on accuracy and decomposed inference speed, while LayerRoute and LayerDrop provide modest speedups but weak accuracy. Interpretation is limited by the audit’s narrow models, tasks, sample size, timing protocol, and deployment scope.

Main findings: SWIFT is stronger than ConfLayers on both accuracy and decomposed speed, while ConfLayers is dominated across all seeds and cells.This is the main finding among periodic-step, search-based methods.
Supplemental analysis: LayerRoute and LayerDrop show modest genuine speedups but are not competitive on accuracy under the verified supplemental protocol.Their coarser, less input-adaptive regime distinguishes them from the primary comparison.
Limitations: The audit covers two instruction-tuned model scales and two tasks, evaluates 100 samples per task, and uses three seeds without power analysis for confidence-interval width.These choices constrain statistical and generalization claims.
Limitations: Timing omits warm-up iterations, while memory footprint, integration complexity, distribution-shift robustness, and quantization compatibility remain outside scope.The audit focuses on accuracy and measured inference/search costs rather than the full deployment profile.
Reproducibility: The released protocol, harness, ports, checkpoints, and per-query files are intended to support rigor-matched efficiency comparisons.Figures use three fixed seeds, and the SWIFT and TinyLlama ports were verified against upstream implementations.

A ARCHITECTURE DIAGRAMS

The architecture diagrams show how ConfLayers and SWIFT make online decisions, while LayerRoute and LayerDrop use trained or fixed routing. Supporting tables document seed-level accuracy for the main comparison.

Main comparison diagrams: ConfLayers thresholds a per-layer confidence estimate against τ, with τ discovered online through Bayesian optimization.The figure distinguishes pure inference cost from search overhead.
Main comparison diagrams: SWIFT self-drafts by skipping a searched subset of its own layers, generates candidates with tree attention, and verifies them in one full-model pass.The diagram represents genuine self-speculative decoding.
Supplemental diagrams: LayerRoute combines a frozen backbone layer, per-layer LoRA adapter, and straight-through hard gate to select adapted output or a residual skip.This supplemental method makes routing decisions through a learned gate head.
Supplemental diagrams: LayerDrop trains robustness to stochastic layer removal, then applies one fixed training-selected pruning pattern identically to every input.It has no router or per-input adaptivity at inference.
Accuracy tables: Table 2 reports main-comparison accuracy as three-seed means with observed ranges, while Tables 6 and 7 provide each seed’s GSM8K and CNN/DailyMail values.The tables separate aggregate reporting from per-seed results.

B.2 SUPPLEMENTAL ANALYSIS: LAYERROUTE AND LAYERDROP

The supplemental analysis provides per-seed accuracy tables for LayerRoute and LayerDrop across GSM8K and CNN/DailyMail under the Section 7.2 protocol.

Supplemental results: Tables 8 and 9 provide per-seed GSM8K exact-match and CNN/DailyMail ROUGE-L values for the supplemental analysis.These values underlie Table 4 and were measured under the verified Section 7.2 protocol.