PaperlayerView exact revision on arXiv

Source-linked AI summary

Weakening Neurons: An Input-Output Functionality in Transformers with Outsize Influence

Sebastian Gerstner, Hilal AlQuabeh, Kentaro Inui, Hinrich Schütze

arXiv:2609.18612v1cs.LGcs.CL

TL;DR

The paper addresses limited input-output analysis of gated neurons by classifying them through cosine similarities between input and output weights. Across LLMs, it finds layer-wise patterns and that weakening neurons, despite being few, influence behavior even at negative gate values.

  • Problem

    Prior work had not developed an input-output taxonomy for gated activations or systematically analyzed how input-output-defined neuron classes are distributed across modern LLMs and affect behavior.

    Source: §1 · p. 1

  • Method

    The paper analyzes gated neurons by using cosine similarities between their input and output weights to define input-output functionality classes.

    Source: §1 · p. 1, §4.2.2 · p. 4

  • Results

    Early-middle layers contain many conditional strengthening neurons and late layers contain weakening neurons; weakening neurons influence model output even at negative gate values.

    Source: §1 · p. 2, §9 · p. 8

  • Takeaways & Limitations

    Input-output functionality is broadly similar across current models, predicts activation frequency, and identifies weakening neurons with large behavioral influence.

    Source: §10 · p. 9

    Ablations show effects on next-token predictions but not the intermediate steps leading from neuron activations to model output.

    Source: Limitations of ablation experiments · p. 10

Abstract

from arXiv · show

We analyze the learned input-output behavior of GLU-based neurons in large language models (LLMs). We propose a simple analysis method: For each neuron, we compute the cosine similarities between its input (reading) and output (writing) weight vectors. In this scheme, a strong negative cosine similarity indicates the neuron weakens the direction it detects in the residual stream, so we call this a weakening neuron. This allows us to gain a number of novel insights. First, we show that nine different LLMs have similar patterns: weakening neurons appear mostly in late layers whereas their counterparts, (conditional) strengthening neurons, are frequent in early-middle layers. Second, we find that weakening neurons display surprising behavior: even though there are few, they activate often and have a large influence on model behavior. Third, weakening neurons have a strong effect on model output when gate values are negative -- which is surprising since negative gate values are not expected to encode functionality.

1 Introduction

The paper addresses the limited input-output analysis of gated neurons, introducing a weight-cosine taxonomy and finding consistent patterns across nine LLMs. A small class of weakening neurons activates frequently and strongly influences model behavior, including at negative gate values.

Motivation

  • Prior neuron analyses often examine activation contexts or output directions, but do not fully capture how a neuron's read and write directions relate.Earlier work considered negative input-output weight cosines, but had not developed a taxonomy for gated activations or systematically analyzed the distribution and behavioral effects of these classes.Source: §1 · p. 1

Contributions

  • The paper introduces an input-output taxonomy for gated neurons based on cosine similarities between their weight vectors.The input weight reads a residual-stream direction and the output weight writes to it; strong negative alignment means the neuron weakens that direction.Source: §1 · p. 1
  • Across nine LLMs, early-middle layers contain many conditional strengthening neurons, while weakening neurons tend toward later layers and can influence output even at negative gate values.Weakening neurons are relatively few but activate often and have a disproportionately large effect on model behavior.Source: §1 · pp. 1–2

2 Related work

Prior interpretability work has examined neurons, features, and input-output behavior, but the paper distinguishes its weight-based analysis of gated-neuron functionality from those approaches. It also situates its analysis alongside research on erasure, suppression, and confidence regulation.

Neuron and feature interpretation

  • Sparse-autoencoder research offers a feature-based alternative to individual-neuron analysis, while sparse combinations of neurons can include single-neuron cases.The cited discussion notes both critiques of SAEs and recent work identifying meaningful neuron classes.Source: §2 · p. 2

Input-output neuron analysis

  • Prior neuron classifications use activation contexts or output weights, while this paper analyzes the mathematical relation between input and output weights in gated neurons.Gurnee et al. (2024) compute weight cosines for non-GLU models but do not interpret their results; gated analyses must also account for the gate.Source: §2 · p. 2

Input-output perspectives

  • Prior input-output perspectives include semantic neuron interpretations and weight-based analyses of attention-head circuits, rather than this paper's neuron-level weight-cosine taxonomy.The paper frames its analysis as a basic mathematical input-output perspective on neuron weights.Source: §2 · p. 2

Suppression and erasure

  • Research on memory management and erasure describes components removing or overwriting earlier representations, including sparse sets of erasure neurons.Rushing and Nanda (2024) hypothesize a relationship between erasure neurons and Gurnee et al. (2024)'s output-based suppression neurons.Source: §2 · p. 2

Confidence and calibration

  • Confidence-related work studies suppression, entropy, token frequency, and calibration, but the cited works do not relate confidence to input-output cosine similarities.The paper hypothesizes a link between confidence and those similarities.Source: §2 · pp. 2–3

3 Gated activation functions

Gated activation functions use two input-side weight vectors and an element-wise gate, unlike traditional activations described here with one input-side matrix. This structure allows gated neurons to produce positive or negative outputs.

Gated activations

  • Unlike traditional activations described here with one input-side weight matrix, gated activations such as SwiGLU use two input-side matrices and element-wise multiplication.GEGLU replaces Swish with GELU while retaining the rest of the described structure.Source: §3 · p. 3

Neuron-level view

  • A SwiGLU neuron adds a scaled output-weight direction to the residual stream, with its activation determined by gate and input projections.The projections are xgate = ⟨wgate, x⟩ and xin = ⟨win, x⟩; xpost is the neuron's activation.Source: §3 · p. 3

Gated activations

  • Gated activations can be negative: when xgate > 0 and xin is strongly negative, the SwiGLU output is strongly negative.Source: §3 · p. 3

4 Method

The method analyzes how neurons read from and write to the residual stream using weight-vector cosine similarities, then classifies their input-output functionality. It applies this framework to gated neurons and compares observed cosines with random baselines.

4.1 Approach

  • The approach treats a neuron as reading residual-stream directions through its input and gate weights and writing a multiple of its output direction back to the stream.The semantic interpretation of detecting and writing concepts is offered as intuition, not a required assumption.Source: §4.1 · p. 3

4.2 Taxonomy of IO functionalities

  • Input manipulation describes neurons whose output direction aligns with a direction detected in the input, whereas orthogonal-output neurons write elsewhere.The taxonomy considers prototypical alignments among weight vectors.Source: §4.2 · p. 3

4.2.1 Taxonomy for non-GLU models

  • For non-GLU models, strongly negative input-output cosine defines weakening neurons, while strongly positive cosine defines strengthening neurons.Source: §4.2.1 · p. 3

4.2.2 Taxonomy for GLU models

  • For GLU models, conditional strengthening or weakening depends on the gate direction, while proportional-change neurons write along the gate direction in proportion to the input direction.The taxonomy also defines strengthening and weakening cases when all three weight vectors are roughly collinear.Source: §4.2.2 · p. 4
  • For cases beyond prototypical alignments, the method classifies neurons by the closest prototype using a ±0.5 threshold, plots marginal cosine distributions, or uses three-cosine scatter plots.Atypical labels mark cases where the cosine between input and gate weights does not match the other similarities.Source: §4.2.2 · p. 4

4.3 Random baselines

  • The analysis tests whether weight-vector cosines differ from random using i.i.d. Gaussian vectors and layer-specific mismatched-cosine baselines; the plotted median cosine shifts from positive early to negative late across nine models.Figure 2 plots median cos(win, wout) by layer for nine GLU models of 2B to 9B parameters.Source: Figure 2 · p. 4, §4.3 · p. 4

4.4 Implementation

  • The implementation uses TransformerLens v2 with preprocessing intended to make weights more interpretable without changing model behavior, plus an additional custom preprocessing step.The authors publish their code.Source: §4.4 · p. 4

5 Where to find weakening neurons

Across GLU-based models, conditional strengthening is common in early-middle layers, while late layers contain more weakening neurons. The section also reports input manipulation beyond the principal weakening/strengthening pattern.

5.1 Non-GLU models

  • The authors apply their method to 24 non-GLU transformer models, but defer those results to the supplementary material.The non-GLU set spans encoder-decoder, encoder-only, and decoder-only architectures, ReLU and GELU activations, and language and non-language training data.Source: §5.1 · p. 5

5.2 GLU models

  • The authors apply their method to 12 GLU-based LLMs, covering both SwiGLU and GEGLU activation functions, and refer readers to Section C.2 for the full model list.Figure 3 shows IO class distribution across layers; Figure 4 plots neuron distributions in selected layers.Source: §5.2 · p. 5
  • Input manipulators (neurons that are not orthogonal output neurons) constitute 25% of all neurons and as much as 50% in early-middle layers 7–11, using zero-based indexing.Neurons classified as orthogonal output often have weight cosine similarities above the significance threshold.Source: §5.2 · p. 5
  • Across all models, early middle layers are dominated by conditional strengthening; in Llama, more than 80% of input manipulators belong to this class.In late layers, weakening neurons are a large share of the relatively few input manipulators, and proportional change is another important class.Source: §5.2 · p. 5
  • Across models, conditional strengthening dominates early-middle layers, while late layers contain more weakening neurons.The strengthening-then-weakening pattern appears across models, with the median read-write cosine falling slightly below zero in late layers of the nine larger models.Source: §5.2 · p. 5

6 Weakening neurons activate often

The section examines whether GLU neuron classes differ in activation frequency, measured by positive gate values. Weakening neurons activate often, and read-write cosine is strongly negatively correlated with activation frequency in OLMo.

6 Weakening neurons activate often

  • The OLMo-7B heatmap plots cos(win, wout) against activation frequency, and the authors describe weakening neurons as disproportionately important despite their small class size.The darker regions represent more neurons.Source: Figure 5 · p. 6
  • The correlation between cos(win, wout) and activation frequency is −0.91 for OLMo, with the relationship mostly due to middle layers.The reported correlation is stronger than the up-to −0.69 correlations reported by Gurnee et al. (2024).Source: §6 · p. 6
  • Conditional strengthening neurons activate very rarely, while conditional weakening neurons activate very often.Here, activation frequency means how often the gate value is positive.Source: §6 · pp. 5–6

7 Ablation experiments

Ablation experiments find that weakening neurons have the strongest effects among tested IO classes, including on output entropy. Conditional ablations locate much of the entropy-sharpening effect in two activation quadrants, while a case study finds an indirect, cumulative effect from a small subset of neurons.

7.1 Effect size of ablating different IO classes

  • Ablating weakening neurons has the highest effect on several tested metrics compared with other IO classes or same-layer random neurons.The experiments compare class-based ablations with the same number of random neurons from the same layers.Source: §7.1 · p. 7
  • Weakening neurons often sharpen the output distribution when ablated, even though they also flatten it in some cases.The authors report the sharpening effect occurs more often than the flattening effect; other classes do not have as large an entropy effect.Source: §7.1 · p. 7

7.2 Conditional ablations

  • Activation quadrants (ii) and (iii) account for a large part of weakening neurons’ entropy-sharpening effect.Quadrant (ii) has xgate > 0 and xin < 0; quadrant (iii) has xgate < 0 and xin < 0.Source: §7.2 · p. 7
  • Quadrant (iii) has a strong effect despite its negative gate activations being relatively rare and small; the authors report that negative gate values affect model behavior, not just training.They state this finding means Swish is not reducible to ReLU for mechanistic interpretability research.Source: §7.2 · p. 7

7.3 Case study of entropy reduction

  • In the studied case, quadrant (iii) activations sharpened the distribution by boosting the correct next token by up to 12 points relative to the ablated run.No single weakening neuron produced the observed effect alone; a subset of eight did, and the effect was indirect and cumulative.Source: §7.3 · p. 8
  • The authors leave a fuller understanding of the mechanisms behind the indirect, cumulative entropy reduction for future work.This limitation applies to the particular case study described.Source: §7.3 · p. 8

8 Case study of a weakening neuron

The case studies contrast a straightforward strengthening neuron with a more complex weakening neuron. The weakening neuron can reinforce an already-present continuation under negative gate values, while its broader behavior remains harder to interpret.

8 Case study of a weakening neuron

  • Strengthening neuron 28.4737 promotes “review” when the residual stream already indicates it as the next token.The authors characterize this as straightforward input-output behavior.Source: §8 · p. 8
  • Weakening neuron 31.9634 has weights indicating it writes “again” when the residual stream contains “minus again,” but its strong-activation examples lack an obvious semantic relation to “again.”The authors describe this neuron as harder to interpret than the strengthening example.Source: §8 · p. 8
  • When negative xgate and xin values produce positive activation, neuron 31.9634 reinforces “again” when it is already a plausible continuation.The authors say this specific case echoes the relevance of negative-gate activations to model behavior found in Section 7.2.Source: §8 · p. 8

9 Discussion

The discussion interprets layer and activation-frequency patterns while emphasizing weakening neurons’ late-layer influence and surprising gate-dependent behavior.

9 Discussion

  • Across investigated LLMs, early-middle layers contain many conditional strengthening neurons, late layers contain weakening neurons, IO classes track activation frequency in most layers, and weakening-neuron ablation has greater output impact.Source: §9 · p. 8
  • The authors argue memory-management and copy-suppression accounts explain negative, but not positive, cosine similarities.Source: §9 · p. 9

9.1 Middle layers, conditional strengthening neurons

  • Middle layers have many conditional strengthening neurons, with cos(win, wout) negatively correlated with positive-gate frequency; confidence management is a proposed interpretation.The proposed mechanism concerns confidence in hidden representations, not next-token predictions.Source: §9.1 · p. 9

9.2 Late layers, weakening neurons

  • Weakening neurons are few yet consistently late and impactful under ablation; small, rare negative-gate activations appear important and can switch toward strengthening-like behavior.Source: §9.2 · p. 9
  • For xgate < 0 and xin < 0, quadrant (iii) shows strengthening-like behavior.Source: §9.2 · p. 9
  • For positive gates, authors hypothesize error correction when xin > 0, but conflict resolution or prediction sharpening when xin < 0.These explanations are speculative rather than conclusive.Source: §9.2 · p. 9, §9 · p. 9
  • Quadrant (iv) activations are small and rare and seem to have little influence when ablated.Source: §9.2 · p. 9

10 Conclusion

The conclusion presents input-output analysis of neurons as a methodology and summarizes findings across current models. It highlights weakening neurons’ influence, including for negative gate values, and points to interpretability research as a potential use for the analysis.

10 Conclusion

  • Weakening neurons greatly influence model behavior, including for negative gate values previously thought not to impact transformer functionality.Source: §10 · p. 9
  • The paper proposes a methodology for analyzing neurons’ input-output behavior.Source: §10 · p. 9
  • The authors hope input-output functionality analysis becomes part of the interpretability toolkit.Source: §10 · p. 9

Limitations

The weight-cosine approach is simple but limited in scope: it complements rather than replaces other neuron analyses, and weight alignment is not a one-to-one measure of semantic similarity. Ablation experiments identify effects on next-token predictions but do not reveal intermediate steps, while conditional ablations and model coverage impose further limits on interpretation and generalization.

Limitations of weight cosine approach

  • The weight-cosine approach is inherently limited in scope, complements rather than replaces other neuron analyses, and does not map weight similarity one-to-one to semantic similarity.The authors report cases where close-to-orthogonal vectors represent very similar concepts.Source: Limitations of weight cosine approach · p. 10

Limitations of ablation experiments

  • Ablations show effects at the next-token prediction level, not the intermediate steps from neuron activations to model output.Conditional ablations can be brittle across layers because ablating an upstream neuron can change a downstream activation’s sign and whether it is ablated; similar effects across conditional ablations do not establish identical underlying reasons.Source: Limitations of ablation experiments · p. 10

Generalizability across models

  • The weight-based analysis was not tested on mixture-of-experts models or models larger than 9B, and compute-intensive corpus experiments used fewer models.Source: Generalizability across models · p. 10

Limited understanding of results

  • The authors’ confidence-management interpretation lacks conclusive evidence, does not explain all findings, and leaves their understanding of weakening neurons limited.They note that this interpretation does not apply to later layers, which contain the majority of weakening neurons.Source: Limited understanding of results · p. 10

G.1 Hypotheses and choice of metrics

This appendix tests proposed neuron-behavior hypotheses, details ablation and neuron-selection methods, and examines individual neuron activations. It also reports that mean ablation can reveal effects missed by zero ablation, and that double checking may enable more precise concept detection.

G.1 Hypotheses and choice of metrics

  • The initial hypotheses were subject enrichment by conditional strengthening neurons and increased entropy from weakening neurons, tested with attribute rate, entropy, loss, correct-token rank, and final-hidden-state scale.Source: §G.1 · pp. 18, 21

G.2 Details on attribute rate

  • The attribute-rate hypothesis was not supported: weakening-neuron ablations had a small, layer-dependent effect, so the authors could not conclude these neurons are responsible for attribute rate.They instead describe weakening neurons as generally important components, not limited to a specific inference stage.Source: §G.2 · p. 21

G.3.1 Method

  • Conditional ablation replaces activation only when its specified gate-and-input condition holds, using the mean activation for cases meeting that condition.Source: §G.3.1 · p. 21

G.3.2 Comparison of mean and zero ablation results

  • Mean ablation recovers effects of weakening neurons that zero ablation misses; the proposed explanation—that activations lie near zero but far from their mean—remains untested.Source: §G.3.2 · p. 21

H Double checking

  • Double checking occurs when approximately orthogonal gate and input vectors have highly overlapping sets of similar meaningful vectors, and may enable more precise concept detection.The authors hypothesize that this narrows the region in model space that activates the neuron positively.Source: H Double checking · pp. 21–22

I.1 Neuron choice

  • Prediction-neuron selection chooses the highest-kurtosis neuron in each discrete read-write class, favoring neurons with interpretable output behavior but not necessarily interpretable overall behavior.Source: §I.1 · p. 22

I.2 Methods

  • The analysis also projects neuron weights into vocabulary space and examines text examples with the strongest positive and negative activations.It saves 16 examples for each activation polarity per neuron.Source: §I.2 · p. 22

I.3 Detailed analysis of weakening neuron 31.9634

  • For neuron 31.9634, negative-gate activations include cases where “again” is already weakly present and the neuron reinforces it; the authors find the positive-gate case less interpretable.Strong positive activations lack a clear pattern; the proposed explanations are hypotheses.Source: §I.3 · p. 22

I.4 Results and analysis for prediction neurons

The case studies show that neurons promoting particular tokens can do so under very different circumstances, with weakening neurons sometimes involving a conflict between an intermediate prediction and the promoted token. Additional analyses report layerwise activation patterns, ablation plots, and non-GLU neuron distributions.

I.4 Results and analysis for prediction neurons

  • Prediction neurons promote specific token sets under different circumstances; weakening neurons can involve an apparent conflict between the intermediate prediction and the token promoted.Source: §I.4 · p. 24

I.5 Results and analysis for prototypical examples

  • The prototypical-neuron examples include a weakening neuron that detects an intermediate “minus by” prediction; negative-gate activations are more interpretable, often occurring when “by” is the next token.Strong positive or negative activations are not particularly interpretable.Source: §I.5 · p. 24

I.6 More case studies

  • The additional OLMo-7B case studies were selected as potentially interesting for nonsystematic reasons, such as strong activation on a named entity.Source: §I.6 · p. 25

J Additional figures and tables

  • The appendix supplies additional activation-frequency, ablation, and layerwise input-output analyses, with further plots available in supplementary material.Source: J Additional figures and tables · p. 25

J.1 Activation frequencies

  • In OLMo-7B’s last layer, activation frequency and input-output cosine have a positive correlation (+0.29), with two clusters differing in cosine and activation frequency.One cluster has cosine around −0.3 and activates very rarely; the other centers around −0.1 to −0.2 and activates a bit more than half the time.Source: §J.1 · p. 25, Figure 10 · p. 26

J.2 Neuron ablations

J.3.1 Non-GLU models

  • Weakening neurons with strongly negative input-output cosine similarities appear in the non-GLU models examined, while strongly positive strengthening neurons are rare.The distribution varies by model; weakening neurons are mostly early-middle layers in GPT2 in the reported examples.Source: §J.3.1 · p. 31
  • For non-GLU models, many neurons have moderately nonzero input-output cosines, and some earlier observations are systematically reported here for the first time.The reported patterns vary across models.Source: §J.3.1 · p. 31

J.3.2 GLU models

The GLU-model results show model-specific waves in conditional neuron classes alongside broader cosine-alignment patterns and tentative observations about neuron directions.

J.3.2 GLU models

  • In Yi, 68% of neurons are conditional strengthening in layer 0, followed by almost none and a second wave of around 25% in layers 11–17.Source: §J.3.2 · p. 36
  • OLMo-1B has a conditional-weakening peak of 1418 of 8192 neurons (17%) in layer 9 of 16, shortly after the conditional-strengthening wave.Source: §J.3.2 · p. 36
  • Most neurons’ cosine similarities remain below 1, possibly reflecting randomness or input-manipulator neurons adding novel information to the residual stream.Source: §J.3.2 · p. 36
  • For most neurons, cos(wgate, win) ≈0; if this is not random, they may operate on two residual-stream directions, increasing expressivity and semantic complexity.Source: §J.3.2 · p. 36
  • Across 12 models, median read-write cosine similarities follow a similar layer pattern, but OLMo-1B switches to negative values earlier than the others.Source: Figure 26 · p. 37

Related papers

Loading 2609.18612v1…