PaperlayerView exact revision on arXiv

Source-linked AI summary

Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters

Yixin Song, Haotong Xie, Zhengyan Zhang, Bo Wen, Li Ma, Zeyu Mi, Haibo Chen

arXiv:2406.05955v2cs.LGcs.CL

TL;DR

Dense LLM inference uses all model parameters, making computational cost a barrier, while common activation functions and straightforward ReLUfication have limited sparsity or may risk performance loss. The paper introduces dReLU-based sparsification and diverse pretraining data, reporting faster inference while maintaining or improving downstream performance.

  • Problem

    Dense LLMs use all parameters during inference, and common activation functions offer limited sparsity while ReLUfication may risk performance degradation.

    Source: §1 · p. 2

  • Method

    The paper introduces dReLU-based sparsification and uses diverse pretraining corpora to improve sparsity while recovering model capabilities.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    2-5× inference speedup is reported, and enhanced models meet or often surpass their originals across downstream tasks.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    dReLU-based sparsification reaches 90% model sparsity while maintaining performance and reducing resource requirements for deployment.

    Source: §8 · p. 10

    Existing shifted-ReLU sparsity gains are limited, while progressive sparsity regularization carries a risk of performance degradation.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Exploiting activation sparsity is a promising approach to significantly accelerating the inference process of large language models (LLMs) without compromising performance. However, activation sparsity is determined by activation functions, and commonly used ones like SwiGLU and GeGLU exhibit limited sparsity. Simply replacing these functions with ReLU fails to achieve sufficient sparsity. Moreover, inadequate training data can further increase the risk of performance degradation. To address these challenges, we propose a novel dReLU function, which is designed to improve LLM activation sparsity, along with a high-quality training data mixture ratio to facilitate effective sparsification. Additionally, we leverage sparse activation patterns within the Feed-Forward Network (FFN) experts of Mixture-of-Experts (MoE) models to further boost efficiency. By applying our neuron sparsification method to the Mistral and Mixtral models, only 2.5 billion and 4.3 billion parameters are activated per inference iteration, respectively, while achieving even more powerful model performance. Evaluation results demonstrate that this sparsity achieves a 2-5x decoding speedup. Remarkably, on mobile phones, our TurboSparse-Mixtral-47B achieves an inference speed of 11 tokens per second. Our models are available at \url{https://huggingface.co/PowerInfer}

1 Introduction

Dense LLMs use all their parameters at inference, creating substantial computational costs; existing activation choices and ReLUfication methods have struggled to deliver high sparsity without risking performance. The paper introduces dReLU-based sparsification and reports stronger downstream performance alongside reduced inference computation and faster generation.

1 Introduction

  • Dense LLMs use all parameters during inference, while common activation functions offer limited sparsity and ReLUfication can risk performance degradation.The paper attributes existing ReLUfication shortcomings to limited sparsity gains and insufficiently diverse or numerous training tokens.Source: §1 · p. 2
  • dReLU addresses negative activations in the GLU component; small-scale models using it match SwiGLU performance at close to 90% sparsity.The training corpora include web, code, and mathematical datasets collected from the open-source community.Source: §1 · p. 2
  • FFNs within MoE experts are also sparsely activated, suggesting an opportunity to combine MoE with ReLU-based sparse activation for faster inference.Source: §1 · p. 2
  • Enhanced Mistral and Mixtral models meet or often surpass their original counterparts across a wide range of downstream tasks.The reported models are TurboSparse-Mistral-47B and TurboSparse-Mixtral-47B.Source: §1 · p. 2
  • TurboSparse-Mistral-7B reaches 90% average FFN sparsity, while TurboSparse-Mixtral-47B rises from 75% to 97%, reducing inference FLOPs.The Mixtral sparsity increase is attributed to incorporating sparse neuron activations.Source: §1 · pp. 2–3
  • 2-5× inference speedup is reported, with up to 10 tokens/s for TurboSparse-Mixtral-47B without a GPU.The introduction also reports an average 2.83× generation speedup.Source: §1 · p. 3

2 Related Work and Background

The background frames efficient LLM inference as a joint algorithmic and systems challenge, with sparse activation requiring both suitable algorithms and hardware-aware exploitation. It reviews conditional computation through MoE routing and ReLU-family neuron sparsity, then introduces the Gated-MLP components used in the study.

2 Related Work and Background

  • Efficient LLM inference combines algorithmic approaches to reduce computation or memory with systems work that aligns hardware and models.The cited examples include model compression, structural changes, speculative decoding, and frameworks such as vLLM.Source: §2 · p. 3
  • Sparse activation requires algorithmic choices such as activation functions and predictors, plus systems methods that exploit sparsity on specific hardware.Prior systems leveraging sparse activation have reported promising inference-efficiency results.Source: §2 · p. 3
  • MoE induces effective sparsity by routing each inference pass to a subset of experts, increasing model capacity without escalating computational expenses.The experts are selected through a trained router subnetwork.Source: §2 · p. 3
  • With ReLU-family nonlinearities, intrinsic activation sparsity lets each neuron be treated as an expert to reduce computation overhead.Source: §2 · p. 3
  • A Gated-MLP applies an activation to the gate projection, multiplies it elementwise by the up projection, then applies the down projection.The activation function may be ReLU or SiLU.Source: §2 · pp. 3–4

3 Analysis

The analysis finds that existing ReLUfication yields only a moderate sparsity gain, motivating dReLU’s changes to both projections. In small-model tests, dReLU maintains competitive performance at high sparsity.

3.1 Limitations about Existing ReLUfication

  • ReLULlama increases sparsity from 40% to 67% over Llama-2-7B, showing the limited gain of existing ReLUfication.Source: §3.1 · p. 4, Table 1 · p. 4
  • After ReLUfication, activations concentrate more around zero because ReLU after the gate projection masks negative values.Source: §3.1 · p. 4, Figure 3 · p. 4
  • Shifted-ReLU’s sparsity improvements are limited, while progressive sparsity regularization risks degrading performance.Source: §3.1 · p. 4
  • Existing ReLUfication leaves the up-projection distribution unchanged, where many values remain negative; masking negative outputs from both projections could increase sparsity.Source: §3.1 · pp. 4–5, Figure 3 · p. 4

3.2 dReLU

  • dReLU applies ReLU after both the up and gate projections before combining their outputs.Source: §3.2 · p. 5
  • dReLU and SwiGLU models show similar convergence; dReLU has lower validation perplexity despite SwiGLU’s lower training loss.Source: §3.2 · p. 5, Figure 4 · p. 5, Table 2 · p. 5
  • At 90% sparsity, dReLU maintains competitive performance, while the SwiGLU model’s perplexity increases severely at 80%.The analysis evaluates sparsity by retaining top-k% activations ranked by absolute magnitude.Source: §3.2 · p. 5, Table 3 · p. 6

4 Are Neurons in Expert still Sparsely Activated?

The section tests whether MoE experts retain sparse activations and finds that MoE performance changes little at sparsity ratios up to 0.5. Mixtral and Mistral also show similar activation distributions concentrated around zero.

4 Are Neurons in Expert still Sparsely Activated?

  • The study examines Deepseek-MoE, Qwen1.5-MoE, and Mixtral, with Llama-2-7B as a comparison.Source: §4 · p. 6
  • Performance drops by about 1%–2% at a sparsity ratio of 0.5, with negligible impact until the ratio exceeds 0.5.The comparison is reported for MoE models in Figure 5(a).Source: §4 · p. 6, Figure 5 · p. 6
  • Mixtral and Mistral have similar activation patterns concentrated around zero, across layers and experts.The authors note that different neurons in the same expert have different functionality and leave further analysis for future work.Source: §4 · p. 6

5 dReLU Sparsification

The section evaluates dReLU-based ReLUfication as a way to recover model performance while increasing sparsity. It applies the change to Mistral-7B and Mixtral-47B, continues pretraining on collected corpora, and then uses SFT.

5 dReLU Sparsification

  • The central question is whether dReLU-based ReLUfication can recover the original model’s performance while achieving higher sparsity.Source: §5 · p. 6
  • The experiments replace the SwiGLU-based FFN with a dReLU-based FFN in Mistral-7B and Mixtral-47B, then continue pretraining.Source: §5 · p. 6
  • Recovery pretraining draws on open-source corpora including web, mathematics, scholarly, general text, and code datasets, with a detailed mixture ratio reported in Table 4.Source: §5 · p. 7, Table 4 · p. 7
  • After pretraining, the models undergo SFT using orca-math-word-problems and bagel.Source: §5 · p. 7
  • Training uses llm-foundry with FSDP parallelism, and the ReLUfication hyperparameters follow empirical results from prior work [69].Source: §5 · p. 7
  • The models are pretrained on 150B tokens using AdamW, with a cosine learning-rate schedule.The reported optimizer settings include β1 = 0.9 and β2 = 0.95; further hyperparameters are listed in Table 5.Source: §5 · p. 7, Table 5 · p. 7

6 Experiments Results

Across downstream benchmarks, the sparsified models match or exceed the stated comparison models while activating fewer parameters. Their layerwise activation sparsity is also high, including within Mixtral’s expert FFNs.

6.1 Downstream Tasks Performance

  • TurboSparse-Mistral-7B outperforms Gemma-2B while activating 3B parameters, and TurboSparse-Mixtral-47B outperforms original Mixtral-47B while activating 4.5B.The evaluations cover OpenLLM Leaderboard tasks and commonsense reasoning tasks.Source: §6.1 · pp. 7–8, Table 6 · p. 8

6.2 Sparsity of Sparsified Models

  • TurboSparse-Mistral-7B has 90% inactive neurons per layer on average, while TurboSparse-Mixtral-47B averages 85% inactive neurons per expert FFN.Mixtral’s expert activation pattern combines with its routing of 2 of 8 experts to yield 3% of parameters activated in each MoE layer.Source: §6.2 · p. 8, Figure 6 · p. 8

7 Practical Inference Speedup Evaluation

The inference evaluation uses predictor-guided neuron-level sparse computation with PowerInfer and compares decoding speed across CPU-only, hybrid GPU-CPU, and mobile settings. Reported speedups range from 2.08–2.28× on CPU-only inference to 22.2× for TurboSparse-Mixtral-47B on mobile against the stated original-model baseline.

7 Practical Inference Speedup Evaluation

  • During SFT, predictors are added for each FFN block, and each Mixtral expert has a predictor that identifies activated neurons when routed.The models are integrated with PowerInfer to measure actual generation speed.Source: §7 · p. 8

7.1 Experiments Setting

  • The experiments compare sparsified models in PowerInfer and PowerInfer-2 with original models in llama.cpp across three hardware configurations.The configurations include two PCs and a OnePlus 12 phone.Source: §7.1 · p. 9

7.2 Pure CPU Inference

  • CPU-only inference yields 2.08–2.28× speedups for the ReLUfied models over the original models.The evaluation is constrained to CPU performance because of DRAM limitations.Source: §7.2 · p. 9, Table 7 · p. 9

7.3 Hybrid GPU-CPU Inference

  • Hybrid GPU-CPU inference yields 2.52–4.64× speedups for the reported models and settings over llama.cpp.Table 8 presents decoding speeds in tokens per second.Source: §7.3 · p. 9, Table 8 · p. 9

7.4 Deploy LLMs on mobile phones

  • 22.2× speedup is reported for TurboSparse-Mixtral-47B on PowerInfer-2 versus original Mixtral-47B on llama.cpp in the mobile evaluation.The authors attribute the gain primarily to PowerInfer-2 exploiting TurboSparse’s high inference sparsity.Source: Table 9 · p. 10, §7.4 · p. 10

8 Conclusion

The conclusion presents dReLU-based sparsification as a way to increase sparsity while maintaining performance and speeding inference. The appendix describes 300M-model training and architecture details.

8 Conclusion

  • 90% sparsity while maintaining performance, with 2-5× inference speedup, is the conclusion's headline for dReLU-based sparsification.Source: §8 · p. 10

A.1 Training Details of 300M models

  • Table 10 reports model accuracy (%) across evaluation datasets and an average; the subsection introduces training details for the 300M models.Source: Table 10 · p. 14, §A.1 · p. 14

A.1.1 Architecture

  • The model adopts an architecture similar to Llama 2 [60], focuses on dReLU and SwiGLU [52], and uses Llama-2-7B attention with prenormalization via RMSNorm [66] and RoPE for positional embeddings.Source: §A.1.1 · p. 14

A.1.2 Training Hyperparameters

Training uses LLaMA-Factory and AdamW, with specified optimizer settings, a cosine learning-rate schedule, and optimization details tabulated in Table 12.

A.1.2 Training Hyperparameters

A.2 Activation Distribution Analysis of MoE Models

The appendix reports that Mixtral's expert FFNs have an activation distribution similar to dense Mistral's. It also points to benchmark metrics for ReLUfied models.

A.2 Activation Distribution Analysis of MoE Models

  • Mixtral's expert FFNs show an activation distribution similar to dense Mistral's, according to the reported analysis.Source: §A.2 · p. 14
  • Figure 7 presents post-activation distributions for Mixtral and Mistral.Source: Figure 7 · p. 15

A.3 Details Performance of ReLUfied Models

  • Table 13 reports common-benchmark results for four models, and the subsection describes detailed ReLUfied-model metrics across commonsense benchmarks.Source: Table 13 · p. 15, §A.3 · p. 15

B Limitation

The authors note that the models have received only 150B continued-training tokens and suggest further training may mitigate capability shortcomings. The broader-impact section connects sparsification with lower resource use and wider access.

B Limitation

  • With only 150B continued-training tokens, the models still have capability deficiencies relative to Llama-3's 15T-token pretraining; the authors expect further training may mitigate them.Source: B Limitation · p. 15

C Broader Impact

  • The broader-impact section frames sparsification as reducing computational and energy demands and widening LLM access for smaller organizations, educational institutions, and researchers.Source: C Broader Impact · p. 15
Loading 2406.05955v2…