The Super Weight in Large Language Models

1 INTRODUCTION

• Pruning one super weight in Llama-7B drops average zero-shot accuracy effectively to zero, while pruning the other top 7,000 outliers affects accuracy by no more than a few percentage points.The single scalar can be less important by magnitude yet far more important to quality than thousands of other outliers.Source: §1 · p. 1
• Super weights occur in early mlp.down proj layers and amplify input inliers into prompt-invariant super activations, which suppress stopword likelihood.The paper attributes the persistence of the super activation at the same magnitude and position to skip connections; pruning the super weight dampens it and shifts logit probability mass to stopwords.Source: §1 · p. 1
• Preserving super outliers makes round-to-nearest activation quantization competitive with SmoothQuant and supports larger weight-quantization block sizes with higher quality.The proposed approach holds out and restores at most a handful of scalar super outliers.Source: §1 · pp. 1–2
• A data-free single-forward-pass method identifies super weights, and the paper provides their coordinates for common open LLMs.The work also describes super weights and super activations together as super outliers.Source: §1 · pp. 1–2

2.1 OUTLIERS IN LLMS

• Prior studies report weight outliers that degrade training loss and downstream performance when disabled, activation outliers that encourage particular attention patterns, and massive activations persisting across layers at a fixed position.Earlier work proposed mitigating massive activations; this paper instead focuses on leveraging them.Source: §2.1 · p. 2

2.2 OUTLIER-AWARE QUANTIZATION METHODS

• Weight-only quantization methods use smaller blocks, channel-wise scaling, clipping, or high-precision storage for sensitive outliers, while mixed-precision arithmetic for many weights can penalize hardware and latency.This paper instead handles at most a half dozen scalars to maintain hardware friendliness.Source: §2.2 · pp. 2–3
• Activation quantization methods rotate, clip, shift, or scale outliers; SmoothQuant requires calibration data, whereas the paper presents a competitive data-free alternative using a small change to round-to-nearest quantization.Source: §2.2 · p. 3
• Where prior work links activation outliers to sensitive weight channels, this paper reports relationships between individual scalars—up to six weights and one activation.Source: §2.2 · p. 3

3 SUPER WEIGHTS

• Pruning one super weight drives zero-shot accuracy to guessing, while pruning 7,000 other outliers has a much smaller effect.The super weight is not necessarily the largest outlier, yet its importance exceeds that of thousands of other outlier weights combined.Source: §3 · p. 3

3.1 IDENTIFICATION OF SUPER WEIGHTS

• Pruning the super weight drastically reduces the massive activation that first appears in its channel immediately afterward, suggesting the weight creates it.The authors call these massive activations super activations.Source: §3.1 · pp. 3–4
• Before down projection, the gate–up Hadamard product creates a relatively large activation that the super weight further amplifies.Source: §3.1 · p. 4
• Super weights can be located by detecting spikes in down-projection input and output distributions using one prompt, then iteratively removing detected weights.The process repeats until large maximum activations are greatly suppressed.Source: §3.1 · p. 4
• The directory covers LLM families and sizes; most examined models have at most three super weights, and Phi-3-mini-4k-instruct has six.Instruction-finetuned models examined retain the same super-weight coordinates as their pretrained counterparts; the directory gives a Hugging Face coordinate example for Llama-7B.Source: §3.1 · p. 4, Table 2 · p. 4

3.2 MECHANISMS OF SUPER WEIGHTS

• Super weights induce super activations that persist throughout the model and suppress stopword likelihood.Source: §3.2 · p. 4

• For Llama-7B, the Layer 2 down-projection input activation’s channel index gives the super-weight row, and the output activation’s channel index gives its column.Source: Figure 3 · p. 5
• Starting after Layer 2, the super activation persists at the same magnitude throughout the model; pruning its super weight decreases that magnitude by 75%.Source: Figure 4 · p. 5, Figure 2 · p. 5
• 49.94 average accuracy after restoring super activations, up from 35.14, salvages approximately 42% of quality loss but does not fully explain the super weight’s influence.The comparison uses the Prune SW and Prune SW,+SA conditions.Source: I II III · p. 5
• For “Summer is hot. Winter is ”, the original model predicts “cold” at 81.4%, while removing the super weight makes “the” the top prediction at 9.0%.Source: I II III · p. 5
• Scaling super weights from 0.0 to 3.0 can improve model accuracy to some extent, with a consistent but miniscule improvement across model sizes.Source: I II III · pp. 5–6, Figure 6 · p. 6
• Across models, removing super weights yields 2-5× larger stopword probabilities; in Llama-7B, “the” rises around 2×, “.” 5×, and “,” 10×.Source: I II III · p. 5, Figure 5 · p. 6

4 SUPER-OUTLIER AWARE QUANTIZATION

• Super outliers are held out to protect inlier quantization, then restored after dequantization to preserve their effects.The approach assumes half-precision arithmetic and applies to both weights and activations.Source: §4 · p. 6

4.1 ACTIVATION QUANTIZATION

• Activation quantization replaces the super activation with the median, quantizes and dequantizes activations, then restores the super activation in FP16.Because the super activation is a single scalar, bitrate and kernel complexity are not significantly impacted.Source: §4.1 · p. 7

4.2 WEIGHT QUANTIZATION

• Prior weight-quantization methods commonly use block sizes of 64 or 128, which add computational and bitrate overhead from half-precision scales and biases.The proposed method addresses large-block INT4 quantization by clipping outliers while restoring the super weight.Source: §4.2 · p. 7
• Weight quantization clips outliers, quantizes and dequantizes the clipped weights, then restores the half-precision super weight.Clipping uses a z-score threshold; the threshold is tuned for minimum reconstruction error using 500 Wikitext-2 training examples.Source: §4.2 · p. 7

5 EXPERIMENTS

• Experiments evaluate LLaMA 7B–30B, Mistral-7B, and OLMo using zero-shot benchmarks plus Wikitext-2 and C4 perplexity.The zero-shot tasks are PIQA, ARC, HellaSwag, Lambada, and Winogrande.Source: §5 · p. 7

5.1 ACTIVATION QUANTIZATION

• On three Llama models, the activation method retains over 70% of SmoothQuant’s improvement over naive quantization on both datasets.It retains above 80% on C4 with Llama-7B and Wikitext with Llama-30B; Table 3 describes the comparison and retained-improvement percentages.Source: §5.1 · p. 8
• Across OLMo, Mistral-7B, and Llama-2-7B, super-activation handling consistently outperforms naive W8A8 quantization on all evaluated models and datasets.SmoothQuant is not reported for these models; OLMo’s non-parametric LayerNorm makes it incompatible with SmoothQuant.Source: §5.1 · p. 8

5.2.1 EXISTING WORKAROUNDS FOR THE SUPER WEIGHT

• AWQ scales the Llama-7B super weight by a factor of 12, while SqueezeLLM’s retained top 0.05% of outliers includes super weights.These methods use different approaches to protect important weights.Source: §5.2.1 · pp. 8–9

5.2.2 SCALING UP BLOCK SIZES

• Super weight-aware quantization degrades less than traditional round-to-near quantization as block size increases, and supports larger blocks with less quality loss.The larger blocks yield lower average bitrate and smaller files; Figure 7 illustrates the block-scaling result.Source: §5.2.2 · p. 9, Figure 7 · p. 9

6 CONCLUSION

• Pruning the super weight destroys text generation, while retaining it can significantly improve quantized-model quality.The conclusion frames both the super weight and its induced super activation as critical super outliers.Source: §6 · p. 9
• Super weights and their induced super activations are the two super-outlier types emphasized as important to model behavior and quality.The conclusion connects their influence to practical detection and management strategies.Source: §6 · p. 9
• The paper presents strategies for detecting and managing super outliers and shares a directory of super weights to support further research.The authors hope the directory will spur study of super-weight properties and implications.Source: §6 · p. 9

A.1 SUPER WEIGHT SENSITIVITY

• Winogrande and Lambada improve consistently with super-weight amplification, while PIQA accuracy is unchanged or slightly lower.The reported patterns cover FP16 models and are also reflected in minor improvements for 4-bit quantized Llama-7B.Source: §A.1 · p. 13, Table 5 · p. 13, Table 6 · p. 13
• On 4-bit quantized models, amplifying super weights produces a small but consistent improvement in average zero-shot accuracy.The reported block sizes are 8x8 and 64x64.Source: Table 6 · p. 13
• When datasets are reduced to guessing accuracy, the average across zero-shot datasets is around 30%.The sensitivity graph displays results starting from zero.Source: §A.1 · p. 13, Figure 8 · p. 13

• Outlier “sparks” in down-projection activation visualizations indicate the row and column indices of super weights.Source: §A.2 · p. 13
• The visualizations track maximum-magnitude activations across all transformer layers for Mistral-7B and OLMo-7B.Source: Figure 9 · p. 14, Figure 10 · p. 14

A.3 LOGIT DISTRIBUTION WITH SUPER WEIGHT REMOVAL

• Stopwords are amplified and non-stopwords see a drastic decrease in likelihood after removing the super weight from Llama-7B.Figure 11 shows the output token distribution before and after removal.Source: §A.3 · p. 14, Figure 11 · p. 14

A.4 ADDITIONAL ACTIVATION QUANTIZATION RESULTS WITH SUPER ACTIVATIONS

• Additional activation-quantization results for Llama-2 7B are reported in Table 7, which covers perplexity on Wikitext-2 and C4.Source: §A.4 · p. 14, Table 7 · p. 15

A.5 SUPER WEIGHTS AND ATTENTION SINKS

• Attention sinks persist after super-weight removal, even though model quality is not preserved.Source: §A.5 · p. 14