LayerRoute: Adaptive Layer-Skipping with LoRA-Preserved Quality for Efficient LLM Inference

1 Introduction

• LayerRoute combines per-layer hard-gated skipping with jointly trained LoRA adapters, gate regularization, and biased middle-layer initialization.The regularization and initialization are described as necessary to escape an all-open equilibrium and reach genuine skip behavior.Source: §1 · p. 1
• Across 10 independently seeded runs, the study reports reproducible skip-pattern structure, wall-clock speedup, and quality preservation, while directly checking that routing is input-dependent.The authors also distinguish stable properties from those that do not reproduce reliably.Source: §1 · pp. 1–2
• The hard-gated architecture is jointly trained with LoRA adapters in under 7 minutes on a single A100.Source: §1 · p. 2
• The paper directly verifies that the trained router performs substantial per-input computation rather than acting as a fixed pruning pattern.Source: §1 · p. 2
• The authors report a systematic diagnostic methodology for investigating what signals drive router decisions, including negative results.Source: §1 · p. 2

2 Related Work

• Early-exit methods use intermediate confidence or calibrated thresholds, while SkipBERT learns input-dependent whole-layer skipping and LayerDrop enables post-hoc depth reduction.DeeBERT [Xin et al., 2020], PABEE [Zhou et al., 2020], CALM [Schuster et al., 2022], SkipBERT, and LayerDrop [Fan et al., 2020] are cited as examples.Source: §2 · p. 2
• LayerRoute shares a skip-connection mechanism with Mixture of Depths [Raposo et al., 2024] but uses joint LoRA fine-tuning and sequence-level rather than token-level routing.Source: §2 · p. 2
• The straight-through estimator [Bengio et al., 2013] is one approach to hard discrete decisions; Gumbel-softmax [Jang et al., 2017] and the concrete distribution [Maddison et al., 2017] are alternatives not explored here.Mixture-of-experts methods such as Switch Transformer [Fedus et al., 2022] and GShard [Lepikhin et al., 2021] route among parallel experts rather than skipping sequential depth.Source: §2 · p. 2
• LoRA [Hu et al., 2022] adapts weights through low-rank updates, and LayerRoute applies it to attention projections while training per-layer routers jointly.Source: §2 · p. 2
• LayerRoute is described as complementary to decoding, memory, kernel, quantization, and structural-pruning techniques because it reduces active layers per input.The passage names speculative decoding [Leviathan et al., 2023, Stern et al., 2018], PagedAttention [Kwon et al., 2023], FlashAttention [Dao et al., 2022], and structural pruning [Ma et al., 2023, Sun et al., 2024].Source: §2 · pp. 2–3
• Although training data draws on agentic workflows [Yao et al., 2023, Schick et al., 2023], the authors do not claim router decisions specifically track task type.Source: §2 · p. 3

3.1 Architecture

• Each of the 24 blocks receives a lightweight per-layer router and LoRA adapters on its attention projections.The routers total 21,528 parameters; the rank-8 LoRA adapters total 1,081,344 parameters.Source: §3.1 · p. 3

3.2 Straight-Through Estimator

• The straight-through estimator uses hard gates in both forward passes while routing gradients through the corresponding continuous sigmoid values.At inference, blocks with gate value 0 are not computed, enabling genuine computation skipping rather than computing and discarding their outputs.Source: §3.2 · p. 3

3.3 Training Objective

• The joint objective adds gate regularization with λ = 1.0 to autoregressive cross-entropy, penalizing uniformly high soft gate values.The regularizer is intended to prevent collapse to an all-open equilibrium.Source: §3.3 · p. 4

3.4 Biased Initialization

• Layer-position-dependent bias initialization breaks the symmetry of gates initialized near 0.5 by starting middle layers below threshold.Those middle layers skip from step 1, exposing their contribution to the language-model loss gradient.Source: §3.4 · p. 4

Base model

• Qwen2.5-0.5B-Instruct is the 24-block, hidden-size-896 backbone used in the experiments.It has grouped-query attention with 14 query heads and 2 KV heads, a SwiGLU feed-forward network, and a vocabulary of 151,936.Source: §4 · p. 4

Training data

• The mixed agentic training dataset contains 10,749 training and 1,194 validation samples from tool-call and reasoning/planning sources.The named sources are Hermes Function Calling v1, Glaive Function Calling v2, GSM8K, and Turing Open Reasoning.Source: §4 · p. 4

Training

• Ten independently seeded runs use 3,000 training steps and take 382 seconds on a single A100 40GB.Each run seeds weight initialization, dropout, and batch ordering; evaluation sample selection is separately seeded per run using the training seed.Source: §4 · p. 4

Evaluation

• Each seed is evaluated on 100 held-out samples, measuring real wall-clock inference latency, gated-model versus full-layer-baseline perplexity, and gate non-triviality.The samples comprise 50 from each of two evaluation sources representing tool-call-style short structured generation and longer-form reasoning.Source: §4 · p. 4

5.1 Skip-Pattern Structure Is Reproducible

• Layers 8–16 were skip-eligible in every seed, while layers 0–7 and 17–23 remained open, with across-seed standard deviation below 0.004 at every layer.The converged mean gate values were 0.27–0.28 in layers 8–16 and 0.72–0.73 elsewhere.Source: §5.1 · p. 4, Table 1 · p. 5

5.2 Genuine, Measured Wall-Clock Speedup

• 1.02×–1.06× speedup (mean 1.04×) across all ten seeds was measured against a genuine full-model baseline with real inference-time skipping.The baseline forced every gate open and was verified to execute all 24 layers.Source: Table 2 · p. 5, §5.2 · p. 5

5.3 Quality Preservation via Joint LoRA Adaptation

• Across all ten seeds and both evaluation splits, gated-model perplexity was lower than the full-layer baseline’s using the same jointly trained LoRA weights.Table 3 reports the gated-minus-full perplexity delta; negative values indicate the gated model outperforms the baseline.Source: Table 3 · p. 5, §5.3 · p. 5

5.4 The Router Is Non-Trivial

• In the nine skip-eligible layers, the input-dependent weight term changed actual skip/run decisions for 87–100% of held-out samples.Outside that band, the weight term rarely changed decisions, consistent with gates already being decisively open.Source: §5.4 · p. 5, Table 4 · p. 6

• Across ten independent runs, LayerRoute reliably reproduces the skip-pattern architecture, positive wall-clock speedup, and quality preservation via joint LoRA adaptation.The authors directly verify that the router’s input-dependent weight term substantially and consistently changes realized computation.Source: §6 · p. 6
• The authors investigated candidate signals but do not claim a specific interpretable signal for skip-eligible-layer decisions because their evidence does not meet their stated rigor threshold.The investigation included linear probes, gradient analyses, an explicit-supervision ablation, and feature correlations.Source: §6 · p. 6

• The router’s per-input decisions are verified as real, but the specific signal they track is unknown and is not the training-data source split.Sequence length correlates more strongly with decisions than source labels, but their independent contributions could not be statistically separated with confidence.Source: §7 · p. 6
• Experiments use Qwen2.5-0.5B only, so generalization of skip patterns, speedup magnitude, or router behavior to larger models is untested.Source: §7 · p. 6
• All experiments use the Qwen2.5 architecture, leaving generalization to other model families untested.Source: §7 · p. 6

8 Conclusion

• Across 10 seeds, LayerRoute reliably produces a consistent skip-pattern architecture.The evaluation protocol separates stable properties from run-to-run variance.Source: §8 · pp. 6–7
• 1.02–1.06× wall-clock speedup and quality preservation via joint LoRA fine-tuning hold across every seed tested.The router also performs substantial, non-trivial per-input computation.Source: §8 · p. 7
• The specific input property driving the router’s decisions remains unconfirmed despite testing and ruling out several natural candidates.The authors report this uncertainty alongside the positive results.Source: §8 · p. 7

A Architecture Diagram

• A hard straight-through gate selects either the adapted layer output or a pure residual skip.The diagram combines a frozen backbone layer, per-layer LoRA adapter, and gate head.Source: A Architecture Diagram · p. 9
• The per-layer router computes its gate from the mean-pooled hidden state entering that layer.Its input-dependent weight term changes realized skip/run decisions for 87–100% of held-out samples in the skip-eligible band.Source: Figure 1 · p. 10
• Layers 8–16 become skip-favored in all 10 seeds, while layers 0–7 and 17–23 remain execute-favored.The router's decisions change actual skip/run outcomes for held-out samples within the skip-eligible band.Source: Figure 1 · p. 10