PaperlayerView exact revision on arXiv

Source-linked AI summary

Model Casting and Low-Parameter Gating: Towards More Sparsely Activated FFNs

Maria Lomeli, Antoine Groudiev, Matthijs Douze, Loïc Cabannes, Pierre Emmanuel Mazaré, François Fleuret, Maximilian Beck, Gergely Szilvasy, Naila Murray, Hervé Jégou

arXiv:2609.31975v1cs.LG

TL;DR

FFN inference cost dominates in most settings, while existing sparsity levels can be limited by what a pre-trained model already admits. The paper uses model casting to target activation sparsity and introduces LoPA Gating to reduce gate cost; it reports practical speedups, including 3.31× on GPU at 90% sparsity.

  • Problem

    FFN inference cost dominates model complexity in most settings, and the sparsity available in pre-trained models can constrain inference-time savings.

    Source: §1 · p. 2

  • Method

    Model casting swaps in a sparsity-inducing activation with adaptive ℓ1 regularization, while LoPA Gating uses a lower-cost parameterization of W1.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    At 90% sparsity, custom kernels achieve 1.53× end-to-end CPU and 2.06× FFN GPU speedups with standard gating, rising to 1.88× and 3.31× with LoPA Gating.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Model casting targets sparsity near 90% at a small NLL cost, and LoPA Gating improves the sparsity-acceleration frontier.

    Source: §1 · p. 2, §6 · p. 10

    The FFN analysis considers single input vectors and does not batch inputs.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

This paper introduces model casting, a mid-training recipe that drastically sparsifies the activations within the Feed-Forward Network (FFN) layer. With this strategy, at inference time, we first compute the output of the gating matrix and, thanks to its high sparsity, we avoid computations with the two other matrices, reducing the FLOP count by up to 3x. While this theoretical speedup is an upper bound, model casting translates into significant speedups both on CPU and GPU. We then introduce LoPA Gating, a new FFN design that increases the maximum theoretical speedup. It is a low-FLOPs parameterization of the gating matrix that overcomes the 3x cap by allocating fewer FLOPs and parameters to the gating matrix, compared to the two other FFN matrices that are sparsely activated. We consider two cases: (i) we cast a pre-trained model with a sparsity inducing activation; (ii) we train with LoPA from scratch. In all settings, we significantly outperform existing pruning solutions and regular RELU-fication. For instance, at matched quality, we achieve a 3.2x FLOP speedup with LoPA Casting, against 1.6x at best for competing methods top-p and TEAL. Using dedicated kernels, we achieve an actual 3.31x speed-up on GPU at 90% sparsity, past the 3x ceiling of standard gating; RELU-fication, meanwhile, plateaus below 80% sparsity.

1 Introduction

FFN inference cost dominates in most settings unless contexts are extremely long, motivating work on FFN speed. The paper combines mid-training activation sparsification with a lower-cost gating design and reports practical CPU and GPU speedups.

1 Introduction

  • FFN inference cost is independent of sequence length but dominates complexity in most settings, except when the input context is extremely long.LLMs also contain fixed parametric memory, primarily in FFN layers, alongside sequence-dependent attention memory.Source: §1 · p. 2
  • The mid-training recipe swaps SILU for R-S+ and adds adaptive ℓ1 regularization, making target sparsity controllable and reaching 90% at small NLL cost.The paper presents this strategy as sparsifying FFN blocks without degrading model quality.Source: §1 · p. 2
  • LoPA Gating parameterizes W1 efficiently, lowering the FFN FLOPs floor and enabling further efficiency gains.It is combined with the activation-swap strategy.Source: §1 · p. 2
  • At 90% sparsity, standard gating yields 1.53× end-to-end CPU and 2.06× FFN GPU speedups, rising to 1.88× and 3.31× with LoPA Gating.At 95% sparsity, the GPU speedup reaches 3.90×.Source: §1 · p. 2
  • Model casting also transfers to MoE expert FFNs or routers and to the off-the-shelf Qwen3-1.7B base model.The paper describes these experiments as evidence that the approach is not tied to specific pre-trained models.Source: §1 · p. 2

2 Related Work

Related work addresses activation sparsity, post-hoc sparsification, and efficiency-oriented architectural changes. Model casting differs by explicitly setting desired sparsity, while LoPA Gating targets the cost of the gate itself.

Emergent and trained-in activation sparsity

  • Earlier activation-swap methods leave attainable sparsity determined by the activation, whereas model casting explicitly sets the desired sparsity.Prior work reports much lower sparsity for gated than non-gated FFNs, and stochastic RELU/SILU switching is another preparation approach.Source: §2 · p. 2

Post-hoc sparsification

  • Post-hoc methods exploit sparsity without retraining, but are bounded by sparsity already present in a pre-trained model; an 8B model’s reported critical sparsity is at most 40%.Methods include predictor-based selection, magnitude thresholding, gate thresholding, and transform-then-threshold approaches.Source: §2 · pp. 2–3
  • Existing methods cannot prune the always-on gate, while LoPA Gating removes that cost; predictor-based schemes target RELU models rather than the non-sparse gated FFNs considered here.The paper compares against TEAL, top-p, and RELU-fication.Source: §2 · p. 3

Sparsity and efficiency by architecture

  • LoPA Gating belongs to architecture-based efficiency work but makes the gate cheap rather than making the whole FFN block small.The related work also connects dense FFN gates to MoE-style routing over hidden units, motivating casting for both FFN gates and MoE routers.Source: §2 · p. 3

3 Method

Model casting changes a pretrained model mid-training so it adapts to a new activation or FFN architecture. The method combines sparsity-inducing activation changes with LoPA Gating, which reduces the gating matrix’s compute cost.

3 Method

  • Model casting changes part of a pretrained model’s architecture midway through training and continues training so the model adapts to the change.The three setups are Activation Casting, LoPA Casting, and Architecture Casting.Source: §3 · p. 3

3.1 Preliminaries

  • R-S+ matches SILU for positive inputs while providing RELU-like sparsification for negative inputs, limiting the activation change to the negative half.Activation Casting replaces SILU with R-S+ at a stable-learning-rate checkpoint, then resumes training with a short warm-up and stable phase before the usual decay.Source: §3.1 · p. 3, §3.2 · p. 4
  • In a gated FFN, zeros in A(W1x) are known before W3 and W2 are computed, allowing corresponding rows and columns to be skipped while W1 remains fully computed.With standard gating, the FLOP-ratio is 3/(3 − 2s), ranging from 1 to 3.Source: §3.1 · pp. 3–4

3.2 Activation casting

  • An adaptive ℓ1 penalty steers average activation sparsity toward a target, but increasing λ also raises cross-entropy and degrades generation quality.λ is updated upward when sparsity is below target and downward when above; the paper sets λ0 = 10^-3 and α = 1.03.Source: §3.2 · p. 4

3.3 LoPA Gating

  • LoPA Gating adds a fixed sparse matrix G to low-rank trainable terms U and V, yielding a full-rank W1 that outperforms a UV-only gate at equal FLOPs and parameters.The rank controls compute allocation and can trade cheaper inference for worse validation NLL; the authors use α = 1/8 unless specified and add RMSNorm between low-rank terms for large-scale training stability.Source: Figure 3 · p. 5, §3.3 · p. 5
  • LoPA Casting retains the parameter count by increasing hidden dimension to H′, while double activation applies nonlinearities after W1 and W3 to improve performance relative to Standard Gating.Figure 2’s example uses D = 2560, H = 7168, r = 400, and H′ = 9787 at 90% sparse R-S+ gates.Source: §3.3 · p. 5, Figure 2 · p. 5
  • LoPA Gating reduces the gating matrix’s compute cost through a low-rank parameterization, allowing the FFN FLOP-ratio to exceed Standard Gating’s 3× ceiling.At α = 1/8, the reported ratio is 13.5 for LoPA Gating versus 3 for Standard Gating as sparsity approaches 1.Source: §3.3 · pp. 4–6

3.4 Architecture casting

  • Architecture Casting replaces a pretrained model’s full W1 with an approximating LoPA parameterization, then resumes training with R-S+ and the new architecture.The replacement reduces FFN parameters when hidden dimension H cannot be increased, which the authors say degrades performance as expected.Source: §3.4 · p. 6

4 Experimental Setup

The experiments use Llama3-derived dense decoder-only models and compare casting against a dense baseline under matched pre-training and mid-training compute. A scaling ladder spans models from 30 million to 5.79 billion non-embedding parameters.

4 Experimental Setup

  • Models use a Llama3-derived dense decoder-only architecture with grouped-query attention, RMSNorm prenormalization, RoPE, and document causal masking.The tokenizer is Llama3’s TikToken byte-pair encoding with 128,000 regular and 256 reserved tokens.Source: §4 · p. 6
  • Training uses a 4T pre-training phase followed by mid-training and a T-token cosine decay; the default schedule is 4T pre-training, 4T W+S, and 1T decay.Mid-training uses warm-up, stable, and decay phases, and the WSD schedule quantifies extra tokens needed to match dense-baseline training loss.Source: §4 · pp. 6–7
  • The dense baseline continues from the same pretrained checkpoint with the same mid-training budget and WSD schedule, but without activation swap or ℓ1 loss.This matches the comparison on pre-training and mid-training compute.Source: §4 · p. 7
  • Sparsity and NLL are measured on held-out validation data, with sparsity averaged over FFN gate post-activations A(W1x).The validation NLL uses the same sequences across models.Source: §4 · p. 7

Scaling ladder

  • The scaling ladder contains eleven pretrained models spanning 30 million to 5.79 billion non-embedding parameters, each trained for 8T total tokens.The models are used for mid-training with model casting and LoPA Gating.Source: Scaling ladder · p. 7

5 Results

Model casting reaches higher FLOP speedups than post-hoc sparsification at matched quality, while keeping NLL costs low at high sparsity. Dedicated kernels turn that sparsity into CPU and GPU wall-clock gains, and the approach also extends to MoE models, quantization, post-training, and Qwen3-1.7B.

5.1 Model casting

  • Activation and LoPA Casting with ℓ1 loss reach close to 90% sparsity with below 1% NLL degradation; LoPA Casting enables up to 5.83× FLOP speedups.Activation Casting costs 0.4 points on the Core-12 average at a 90% sparsity target.Source: §5.1 · p. 7
  • At 88.6% activation sparsity, the model incurs +0.083 NLL; after the same mid-training budget, top-1 routing has worse NLL (2.073 vs. 2.064) and no activation sparsity.At λ = 10−3, the model reaches 59.0% activation sparsity for +0.033 NLL.Source: §5.1 · p. 8
  • Quantization-aware training combined with model casting gives a 1.57% NLL degradation in int8, and low-precision kernels achieve an 8.13× speedup at 90% sparsity.LCFT and SFT cost model-casting variants about as much as the dense control.Source: §5.1 · p. 8

5.2 Comparison to prior FFN sparsification methods

  • At matched quality, model casting yields 2–3.2× FLOP speedups versus 1.3–1.6× for top-p and TEAL, and outperforms RELU-fication in sparsity and FLOP speedup.Casting targets around 90% sparsity, unlike RELU-fication, which offers no comparable control mechanism.Source: §5.2 · p. 9, Table 2 · p. 9
  • On Qwen3-1.7B-Base, Activation and Architecture Casting both exceed 89% sparsity, while Architecture Casting has 0.062 higher NLL and removes 294M parameters.Architecture Casting’s NLL is on par with models having equivalent parameter counts.Source: §5.2 · p. 9

5.3 CPU/GPU optimization for FFNs with sparse activations

  • At 90% sparsity, LoPA Gating achieves 1.88× end-to-end CPU and 3.31× FFN GPU speedups; at 95%, its GPU speedup is 3.90×.At 90% sparsity, Standard Gating reaches 1.53× on CPU and 2.06× on GPU.Source: §5.3 · p. 10, Figure 5 · p. 10

6 Conclusion

Model casting makes high activation sparsity a training-time target, and LoPA Gating improves the sparsity-acceleration frontier. The approach extends to Architecture Casting and MoE experts, with up to 3.3× wall-clock FFN speedups and below 1% NLL degradation.

6 Conclusion

  • Model casting reaches activation sparsity closer to 90% at a 1% quality drop, compared with the below-40% estimate for an 8B LLM from Szatkowski et al. (2026).The sparsity level can be set as a target during training.Source: §6 · p. 10

A Ablation on the LoPA Gating design

The ablation favors LoPA’s diagonal and tiled-identity fixed components over tested alternatives, with double activation helping the LoPA parameterization. The accompanying baseline details distinguish the FFN sparsity sites and report trained-in versus post-hoc speedups at matched quality.

A Ablation on the LoPA Gating design

  • Low-rank + Diagonal and Low-rank + Tiled Identity outperform the other tested W1 variants, while double activation substantially reduces NLL for LoPA Gating.Their minor NLL differences are attributed to training noise because the fixed components are identical up to permutation.Source: A Ablation on the LoPA Gating design · pp. 15–16, Table 4 · p. 16
  • The ablation supports LoPA Gating as an efficient W1 parameterization and indicates that the positions of the fixed matrix’s 1s are not meaningful.The design uses fixed components with a single ±1 in each row when approximating a full matrix with a low-rank parameterization.Source: A Ablation on the LoPA Gating design · p. 16

B Baselines: additional details

  • Model casting targets gate-output sparsity and, with double activation, gated-intermediate sparsity; top-p and TEAL instead exploit input sparsity.These sites reduce different FFN matrix multiplications.Source: B Baselines: additional details · p. 16, Table 5 · p. 17
  • At matched reference NLL, casting stays near 2.0× FLOP speedup for Activation Casting and 3.2× for LoPA Casting, while top-p and TEAL fall toward 1.3× with longer mid-training.The comparison reports theoretical FFN FLOP speedup across mid-training budgets.Source: Table 6 · p. 17

C Case study: Mixture-of-Experts

The MoE case study applies model casting to expert FFNs and routing, while fixed-penalty experiments show that sparsifiability varies with model scale. Casting two active experts can outperform a one-expert SILU comparison in both NLL and FLOPs, but FFN casting does not always recover pre-training loss.

C Case study: Mixture-of-Experts

  • Router casting can produce intermediate sparsity–NLL operating points, but a 75% target yields validation sparsity about five percentage points below target.Using λ = 0 is the most competitive approach in validation NLL among the reported router-casting settings.Source: C Case study: Mixture-of-Experts · p. 18, Table 8 · p. 19
  • After 4T of mid-training, MoE FFN Activation Casting does not catch up with pre-training loss, whereas SILU does.The reported comparison is for the MoE setting and the 4T mid-training duration.Source: C Case study: Mixture-of-Experts · p. 19
  • Increasing the MoE FFN penalty raises sparsity by up to 20%, with NLL about 2.6% higher than the λ = 0 MoE.Figure 7 also describes an increase of about 25% at the cost of greater NLL degradation than in the non-MoE case.Source: C Case study: Mixture-of-Experts · p. 19, Figure 7 · p. 20
  • R-S+ at 90% sparsity with two active experts yields lower NLL and FLOPs than SILU with one active expert.The comparison is for MoEs with eight experts.Source: Table 10 · p. 20

D NLL vs sparsity for different fixed lambda values

  • Across model levels 2–10, an ℓ1 penalty can push sparsity beyond 90% with little NLL degradation, though a fixed λ produces scale-dependent sparsity.Without a sparsity-inducing loss, natural sparsity increases by six percentage points from level 2 to level 10; at λ = 0.03, sparsity rises by 10 percentage points between those levels.Source: D NLL vs sparsity for different fixed… · pp. 20–21

E Scaling ladder

  • The scaling ladder specifies each compute level’s parameter count, architecture, peak learning rate, and training budget.Its architecture fields include FFN input and hidden dimensions and layer count.Source: Table 12 · p. 21, E Scaling ladder · p. 21

F Architecture Casting vs Standard Gating at reduced parameter count

Architecture Casting is compared with a dense SILU model matched for FFN parameter count, and its initial NLL gap narrows with more casting time. At 5.79B parameters, LoPA Casting has a 0.65% relative NLL degradation from the dense baseline.

F Architecture Casting vs Standard Gating at reduced parameter count

  • Architecture Casting eventually outperforms a dense SILU baseline at iso-parameters as casting time increases.The dense comparison uses hidden dimension H = 3712 to match the cast models’ FFN parameter count; the initial gap narrows over time.Source: F Architecture Casting vs Standard… · p. 22, Table 13 · p. 22
  • The reported NLL degradation is attributed to parameter and sparsity differences, rather than the architecture change during training.This conclusion follows the iso-parameter comparison of Architecture Casting and the reduced-parameter dense baseline.Source: F Architecture Casting vs Standard… · p. 22, Table 13 · p. 22

G Full results for different casting durations

  • Across casting durations, the gap between cast models and the SILU reference narrows; Architecture Casting starts with a wider gap that quickly reduces.The models are pre-trained for 4T tokens, and the casting phase ends with 1T tokens of learning-rate decay.Source: G Full results for different casting… · p. 22

H Validation NLL for a model with 5.8B weights

  • At 5.79B parameters, LoPA Casting has a 0.65% relative NLL degradation from the dense baseline, smaller than for the 0.81B models.The paper relates this result to larger models being easier to sparsify.Source: H Validation NLL for a model with 5.8B… · p. 22

I Catch up times at mid-training

The section reports catch-up measurements and tests model casting alongside quantization and alternative activations. Combined sparse and low-precision inference reaches substantial FFN speedups, while quantization adds sub-additive NLL degradation and sparsity-only pre-training is not beneficial in the reported ablation.

I Catch up times at mid-training

  • Table 16 reports the first mid-training time each model catches the original NLL, evaluated every 0.25T and rounded up by at most 0.25T.For ℓ1 runs, a 90% target corresponds to about 89% measured sparsity; LoPA uses rank ratio α = 1/8.Source: I Catch up times at mid-training · p. 22, Table 16 · p. 24

J Quantization-aware training

  • For int8, quantization costs 0.013 nats on sparse activation-casting models versus 0.015 nats on the SILU reference, with the same values for LoPA casting.The authors describe the combined NLL degradations as sub-additive, indicating that the methods are compatible.Source: Results · p. 24, Table 17 · p. 24

Results.

  • At 90% sparsity, LoPA in FP32 gives 3.31× FFN speedup; combined with quantization, speedups reach 8.13× for int8 and 8.51× for int4.These measurements are for a 5.79B model and are relative to the full-precision dense baseline.Source: Results · p. 24, Figure 9 · p. 25
  • Adding ℓ1 loss during SILU pre-training yields similar or worse NLL and reduces sparsity after casting, so the reported results indicate that casting only during mid-training is sufficient.The ablation used the same ℓ1 weight during pre-training and casting.Source: Results · p. 25, Table 18 · p. 25

M Casting with other activation functions

  • Among sparsifying alternatives, Shifted RELU reaches 75–79% sparsity at an NLL cost, while R-S+ tracks SILU within a few thousandths and delivers 65–69% sparsity before ℓ1.SILU, G-S+, and GELU give the best NLL but are not sparsifying; squaredRELU has 40% exact-zero sparsity in this comparison.Source: M Casting with other activation… · p. 27, Table 19 · p. 27

N Qwen3 mid-training details

The Qwen3-1.7B appendix reports a learning-rate sweep and mid-training comparisons across activation and sparsity-penalty settings. Separate bfloat16 measurements show higher GPU speedup for LoPA than Standard Gating at 90% sparsity.

N Qwen3 mid-training details

  • The Qwen3 mid-training recipe uses 4T warmup-stable tokens followed by 1T with decay, corresponding to 100 TPP at level 8.The described scaling-ladder unit is T = 10^400 tokens.Source: N Qwen3 mid-training details · p. 27
  • The Qwen3 learning-rate sweep evaluates short-context tasks and validation NLL; the nine-dataset NLL average includes five in-mixture datasets and four out-of-mixture probes.GovReport is singled out as an out-of-mixture probe and is included in All 9.Source: N Qwen3 mid-training details · pp. 27–28, Table 21 · p. 28, Table 20 · p. 28
  • The Qwen3-1.7B mid-training runs differ by FFN activation and sparsity penalty, and the table marks the best trained checkpoint per NLL column.The activation settings include R-S+ without penalty, Activation Casting targeting 90% sparsity, and plain SILU mid-training.Source: Table 22 · p. 29

O GPU speedup in bfloat16

  • At 90% sparsity in bfloat16, GPU speedup is 1.71× for Standard Gating and 2.41× for LoPA Gating versus the standard dense case.Figure 11 measures throughput for a 5.79B model.Source: Figure 11 · p. 29

P Short-context evaluations

Short-context evaluations cover code generation, commonsense and reasoning, and additional knowledge benchmarks for Qwen3-1.7B. Across the scaling-ladder models, casting changes average benchmark scores little; on Qwen3-1.7B, the mid-training mixture has a larger performance cost than casting.

Evaluation benchmarks

  • The short-context suite includes code generation and commonsense or general-reasoning tasks, with MMLU, CSQA, and BoolQ added for Qwen3-1.7B.Those three knowledge and commonsense benchmarks are excluded from the Core-12 average.Source: P Short-context evaluations · pp. 29–31

Models trained from scratch

  • LoPA Casting loses 0.2 points at 75% sparsity and gains 0.1 at 90% on the twelve-benchmark average; Activation Casting loses 0.1 and 0.4 points, respectively.No individual benchmark shifts by more than 2.6 points in either direction.Source: P Short-context evaluations · p. 31

Qwen3-1.7B

  • Qwen3-1.7B-Base scores 65.0 on MMLU, 74.9 on CSQA, and 75.6 on BoolQ, motivating checks for knowledge erosion during mid-training.The level-8 model scores 38.2 on the Core-12 average and is close to chance on harder multiple-choice tasks.Source: P Short-context evaluations · p. 31
  • 2.5 and 8.5 points: the Qwen3-1.7B mid-training mixture lowers Core-12 and knowledge-3 means by those amounts, while casting on top costs 1.6 and 0.7 points, respectively.Activation swap without the ℓ1 loss costs 0.4 on Core-12 while gaining 1.0 on knowledge-3.Source: P Short-context evaluations · pp. 31–32, Table 24 · p. 32

Q Post-training

After post-training, casting variants retain performance close to the dense baseline on the Core-12 average, and show no measurable change in chat behavior. All variants retain measurable instruction-following ability, although the 90% sparsity variants show some degradation.

LCFT and SFT

  • The comparison tests Activation Casting and LoPA Casting at 75% and 90% sparsity against a dense SILU control after LCFT followed by SFT.The evaluation uses the authors’ mid-trained level-8 models.Source: Q Post-training · p. 32
  • After 8,000 SFT steps, 90% casting variants are 0.1–0.2 below the dense baseline, while both 75% variants improve by 0.3.SFT costs every model 2.4 to 3.2 points by step 8,000; LCFT shifts the average by at most 1.0 point.Source: Q Post-training · p. 32, Table 26 · p. 33

Chat-format evaluations

  • Chat evaluations agree across the six variants within 1.6 points on cruxeval_output, 4.0 on gsm8k_chat, and 3.2 on mbpp_chat at step 8,000.The authors report that casting does not measurably change chat behavior.Source: Q Post-training · p. 32

Instruction following

  • All casting variants retain measurable IFEval instruction-following ability after 8,000 SFT steps; LoPA Casting at 75% exceeds the dense baseline.The authors note some degradation for the 90% variants, which could be attributable to the extreme sparsity operating point.Source: Q Post-training · p. 34, Table 27 · p. 34

Related papers

Loading 2609.31975v1…