Not All Layers of LLMs Are Necessary During Inference

• Existing pruning and sparse-model approaches alter LLM parameters and may compromise generalization, motivating inference efficiency that preserves generalization and in-context learning.LLM inference is resource-intensive because of models’ large parameter counts.Source: §1 · p. 1
• The analysis finds that early stopping works and that simpler tasks require fewer inference layers than more complex tasks.The key decision is when to stop adaptively for each input instance.Source: §1 · pp. 1–2
• 17.8% average pruning ratio and up to 43% on sentiment tasks come with nearly no performance drop (<1%) across Llama2 and OPT experiments.The feature-collection and classifier costs are described as negligible compared with LLM inference.Source: §1 · p. 2
• AdaInfer uses statistical features from LLM blocks and SVM or CRF classifiers to predict the optimal layer at which to stop inference.The method uses logits to construct its features.Source: §1 · p. 2
• AdaInfer leaves model parameters unchanged, so the paper states that it does not affect generalization or in-context learning and is orthogonal to other acceleration techniques.Source: §1 · p. 2

• Adaptive inference methods broadly use dynamic depth, which changes layer execution, or dynamic width, which changes channels or parallel subnetworks.Source: §2 · p. 2
• Early exit has been used in vision networks and BERT, and recent work discusses confidence-based early exit for language models [Bolukbasi et al., 2017; Huang et al., 2017; Teerapittayanon et al., 2016; Li et al., 2020; Liu et al., 2020; Li et al., 2021; Kong et al., 2022; Schuster et al. (2022); Varshney et al. (2023)].Source: §2 · p. 2
• AdaInfer applies early exit to decoder-only LLMs and differs from skip-layer methods by tailoring the pruning ratio to each instance without altering model parameters.The paper identifies each block’s logits as crucial for its early-exit classifiers.Source: §2 · p. 2
• Dynamic-width methods include channel reduction, mixture-of-experts structures, and pruning attention heads or output neurons; quantization and sparsity are described as orthogonal acceleration areas.Source: §2 · p. 2

3.1 Preliminary: LLM Building Blocks

• Decoder-only LLMs process inputs layer by layer, making inference complexity linearly related to decoder depth; Llama2-7B has 32 layers and Llama2-13B has 40.The section first reviews LLM components, including embeddings, decoder blocks, and the classification layer.Source: §3.1 · pp. 2–3, §3 · p. 2

3.2 Not all Layers are Necessary

• The experiments compare accuracy at each decoding layer with final-layer accuracy on sentiment analysis and MMLU using Llama2-7B and Llama2-13B.Source: §3.2 · p. 3
• On Llama2-13B sentiment classification, early exit at layer 21 on average (variance 5.1) achieves accuracy comparable to the final layer; simpler examples use about 18 layers and more complex ones about 24.Early stopping also works on Llama2-7B.Source: §3.2 · p. 3
• Sentiment accuracy on Llama2-7B matches the final-layer result by layer 24, while MMLU accuracy tends to improve with deeper layers.The same trend holds across four settings from 0-shot to 3-shot learning.Source: §3.2 · p. 3, Figure 1 · p. 3
• The authors extend earlier findings from vision networks and BERT to decoder-only LLM inference, concluding that the optimal stopping layer varies across tasks and instances.Stopping must therefore be dynamically predicted for each input.Source: §3.2 · p. 3

4 AdaInfer: Adaptive Inferences

• AdaInfer captures signals at each decoding layer and predicts whether to stop, bypassing subsequent decoder layers when the classifier selects early termination.The feature-selection module builds a vector for each input instance at each layer.Source: §4 · p. 4, Figure 2 · p. 4

4.1 Feature Selection

• The feature-selection problem is identifying when a representation is sufficient for a task using features that remain compatible across different LLMs.The paper notes that representations become more fine-grained in deeper layers.Source: §4.1 · p. 4
• Feature options include gap, top probability, and cosine similarities between adjacent blocks’ attention, MLP, and hidden-state values.Source: §4.1 · p. 4
• Gap and top-probability values increase and stabilize in deeper layers, and their activation varies across tasks; the analysis presents them as universal stopping features.These trends are also reported for Llama2-7B, OPT-13B, and GPT-J; cosine-similarity features show subtler layer differences.Source: §4.1 · p. 4, Figure 3 · p. 4

4.2 Classifier

• A classifier receives features for each layer and is trained with positive labels when an intermediate layer predicts the same next token as the final layer.The paper considers SVM, which ignores sequence context, and CRF, which models sequences across decoding layers.Source: §4.2 · pp. 4–5

5 Experiments

• The experiments cover zero- and few-shot question answering and text classification, using OPT and Llama2 backbones spanning 7B–70B parameters and 32–80 layers.Few-shot tests use 5, 10, 15, and 20 examples; reported metrics include top-1 accuracy, pruning ratio, average active layers, and variance.Source: §5 · p. 5, §5.1 · p. 5

5.2 Main Results

• <1% accuracy impact versus dense models, and AdaInfer surpasses the fixed-layer ShortGPT5 and ShortGPT3 settings in the reported comparison.ShortGPTp has a significant performance drop on almost all tasks.Source: §5.2 · p. 6, Table 2 · p. 6
• 17.8% average pruning, with task-level ratios from 9% to 43%; sentiment reaches 43% savings on Llama2-13B, while question answering ranges from 9% to 20%.The varying ratios indicate that AdaInfer selects different early-exit configurations for different task inputs.Source: §5.2 · pp. 6–7
• AdaInfer yields 1.03x speedup on MMLU and 1.17x on Sentiment in the reported batch-size-1 runtime tests.The last-token hidden-state conversion costs 0.03% of total Transformer inference FLOPs.Source: §5.2 · p. 7, Table 3 · p. 7

5.3 Evaluation on Alternative Exit Strategies

• GAP and CRF reduce computational costs by 3% to 50% while maintaining comparable LLM performance; in zero-shot tests, GAP outperforms CRF.GAP uses a stopping threshold of 0.8; CRF is an alternative classifier.Source: §5.3 · p. 7

5.4 Evaluation across Scaling Law

• On Llama2-70B, AdaInfer matches or slightly exceeds dense accuracy in zero-shot tests with 10% to 50% lower computation, but few-shot accuracy drops by 1% to 25%.The authors identify feature engineering for larger models as future work.Source: §5.4 · p. 7

5.5 Generalization Study

• SVM shows satisfactory intra-task and inter-task generalization, whereas CRF can terminate too early and inter-model classifier performance is moderate.The authors select SVM based on the generalization and main-result experiments.Source: §5.5 · pp. 7–8

5.6 Factor Study

• Adding attention features has no discernible impact, while MLP and hidden-state features adversely affect results; the authors conclude logits indicate whether forward progress is sufficient.The reported results align with the trend in Figure 3.Source: §5.6 · p. 8

6 Conclusion

• The authors report that intermediate layers can suffice for some tasks, motivating inference that stops before all LLM layers are evaluated.The analysis precedes the AdaInfer method.Source: §6 · p. 8
• AdaInfer predicts an input-specific stopping layer from the most likely token’s probability and its gap to the next most likely token.The classifier uses easily obtained features and leaves model parameters unchanged.Source: §6 · p. 8
• 17.8% average pruning ratio and up to 43% on sentiment tasks are reported with less than 1% performance drop.Experiments cover Llama2 series and OPT models.Source: §6 · p. 8

Limitations

• Logits from each block are identified as critical features for early-exit classifiers in LLMs.The paper incorporates this finding as a key AdaInfer design choice.Source: Limitations · p. 8
• AdaInfer relies on a single forward pass and has not been extended to sequential generative tasks.The authors describe this as an avenue for future research.Source: Limitations · p. 8
• The authors note that features beyond logits may prove more effective.This is stated as a possible direction, not an established result.Source: Limitations · p. 8

Ethics Statement

• The statement raises possible misuse through injecting or amplifying logits as features to produce undesirable behavior.The concern is framed as a potential risk of enhanced inference capabilities.Source: Ethics Statement · p. 8
• The stated research aim is to optimize large-scale model inference without modifying parameters.The authors associate this aim with efficiency gains and reduced energy consumption.Source: Ethics Statement · p. 8
• The misuse concern is specifically tied to exploiting logits as features in large neural language systems.The statement does not describe a demonstrated misuse event.Source: Ethics Statement · p. 8

A Computation Cost

• Classifier computation is described as negligible compared with large-model inference, despite higher training than prediction complexity for SVM and CRF.SVM prediction is O(d) per inference, while CRF prediction is O(S × M).Source: A Computation Cost · p. 11
• Each Transformer block requires 24Bsh^2 + 4Bs^2h FLOPs, while the language-model classification layer requires 2BshV FLOPs in a forward pass.The passage defines the terms in the computation-cost setup.Source: A Computation Cost · p. 11
• For Llama2 7/13/70B, LM-head conversion accounts for 0.000288, 0.000236, and 0.000152 of total Transformer inference FLOPs, respectively.For OPT 13B, the corresponding fraction is 0.000367.Source: A Computation Cost · p. 11
• For individual tasks, AdaInfer can outperform the dense model in zero-shot or few-shot accuracy, which the authors suggest may reflect deep layers over-representing some tasks.The passage presents this explanation as a possibility.Source: A Computation Cost · p. 13

B Classifier Training

• Each decoder layer yields a feature vector and a probability distribution from its hidden state passed through the classification layer.The distribution is computed as P = softmax(WH + b).Source: B Classifier Training · p. 13
• A layer’s feature vector receives a positive label when its highest-probability token matches the instance label, and a negative label otherwise.An L-layer model therefore yields L feature-label pairs per input.Source: B Classifier Training · p. 13

C More Observation of LLMs

• “Gap” and “top prob” gradually increase during inference, stabilize in deeper layers, and vary across tasks.These trends align with the findings discussed in Section 4.1.Source: C More Observation of LLMs · p. 13

D Comprehensive Summary of Results

• Tables 7 and 8 summarize results for LLMs using different classifiers and highlight the best result for each task in the current setting.The supplied passages do not provide the table cell values.Source: D Comprehensive Summary of Results · p. 13
• Early exits are feasible across different tasks, vary by instance, and achieve accuracy comparable to dense models in zero-shot and few-shot settings.The reported test metric is top-1 accuracy.Source: D Comprehensive Summary of Results · p. 13
• Table 7 covers question answering, while Table 8 covers text classification and rule understanding under the same settings.For question answering, few-shot sample sizes {5, 10, 15, 20} are averaged; AdaInfer uses SVM and the rule variant uses a GAP threshold of 0.8.Source: Table 7 · p. 14, Table 8 · p. 14