Rethinking Neural Nonlinearity as Gating

1 Introduction

• Analog in-memory computing makes MACs nearly free, shifting hardware bottlenecks toward ADC/DAC crossings that dominate chip area and power.The paper proposes a primitive with continuous per-neuron choice, implicit pre-MAC gating, and a universal family of activations.Source: §1 · pp. 1–2
• The usual activation-after-linear-layer pattern is a convention: assigning the activation to each pre-MAC input preserves the network function.For ReLU, each input either passes unchanged or is zeroed.Source: §1 · pp. 1–2
• Threshold gating selects among affine branches for each input element before summation, replacing an explicit nonlinearity with input-conditioned gating.A hard gate is a limiting case, while soft gates extend the primitive to smooth activations.Source: §1 · p. 2
• The paper presents five contributions: a TG primitive, activation equivalences, a minimal branch theorem, softmax conversion, and gains in expressiveness, training, compression, and interpretability.The softmax extension includes lossless conversion on pretrained LLMs.Source: §1 · p. 2

2.1 The Threshold Gating (TG) Primitive

• TG uses soft gates forming a partition of unity to blend affine branches before the unchanged layer operation, with τ controlling softness.The TGS, TGW, and TGA tiers progressively absorb branch constants, coarsen gate sharing, and tie branch weights.Source: §2.1 · pp. 2–3

2.2 Standard Activations as TG Instances

• The cited propositions give ReLU, SiLU, and Tanh exact TG forms with K=2 each; GELU has a K=2 approximation with ≤0.3% element-wise error, and Hardtanh nominally uses K=3.Source: §2.2 · pp. 3–4

2.3 Interpretability and Families of Activations

• Branch roles define multiplicative activations, which carry the input forward, and constant activations, which use it only for gating; calibration reduces GELU residual to within 0.04%.Reading activations through their branch parameters makes their roles explicit.Source: §2.3 · pp. 4–5

2.4 The Minimal Branch Theorem

• For well-behaved activations, K=2 TG can match asymptotic slopes and a single transition, while K=1 cannot represent a strict transition.The stated class has a bounded derivative and at most one local extremum in its derivative.Source: §2.4 · pp. 5–6

2.4.1 Activation Complexity and Trainability

• In the Hayou, Doucet, and Rousseau [6] setting, Lmax ∝1/κ(f)^2; the authors argue K=2 misses only activations difficult to represent and train deeply.A single Sigmoid gate imposes at most one inflection point.Source: §2.4.1 · p. 6

2.5 Conversion of Pretrained Networks

• Δ=0.00% accuracy is reported for exact ReLU, SiLU, Sigmoid, and Tanh conversions; ViT-B/16 moves from −1.09% (generic) to −0.40% (per-layer τ) to 0.00% (K=3).Conversion replaces activation modules without modifying layer weights or requiring retraining.Source: §2.5 · p. 7

Expressiveness

• Training from scratch with TGLs demonstrates gains in width reduction, depth reduction, performance, and training speedup.The experiments use representative architecture–dataset pairs for the four axes.Source: §3 · p. 7

Performance

• 1.2–4.5 WER points of improvement across H∈{128,...,1024} on AN4, with less than 0.5% parameter overhead, using constant-family TGA on DeepSpeech2-LSTM.Releasing the constant-family constraint to multiplicative gates degrades performance back toward baseline.Source: Figure 3 · p. 7, §3 · p. 8

Training speedup

• 1.47–2.04× training-step speedup over the corresponding baseline activation across four model–dataset pairs, using TGW K=2.Speedup is measured as the steps required to reach the baseline’s converged metric; the result averages three seeds.Source: Figure 3 · p. 7, §3 · p. 8

TG-softmax conversion

• Each softmax output is a Sigmoid gate against a threshold θi = LSE(z−i), with the threshold computed by a cross-element reduction.The conversion uses K=2 constant branches and τ=1, enabling TG-form inference while preserving pretrained weights.Source: §4 · p. 8

Forward-attention scope

• TG plus one integration step covers all forward-attention nonlinearity, while algorithmic and hardware implications are left to future work.The paper notes that the integration is a single-stage analog operation.Source: §4 · p. 8

End-to-end conversion

• SmolLM-135M and TinyLlama-1.1B preserve PPL to floating-point precision (∆= 0.00%) after end-to-end TG conversion without fine-tuning or calibration.GPT-2 has +0.22% WikiText-2 drift from its non-exact GELU at K=2, recovered to ∆= 0.00% at K=3.Source: Table 5 · p. 8, §4 · p. 8

Analog implementation

• A MOSFET differential pair implements the tunable Sigmoid gate, while a resistive crossbar computes the linear MAC.The gate steepness sets τ and its input offset programs θ; in the high-gm/Ib regime, it saturates into a step function without a separate comparator.Source: §5 · pp. 8–9

Unified front-end

• A single configurable Sigmoid front-end suffices across layers, eliminating bespoke per-activation circuitry.The circuit can configure desired (τ,θ) sharing granularity at modest per-device storage cost.Source: §5 · p. 9

Validation boundary

• Circuit-level integration and silicon validation are deferred to future work.Source: §5 · p. 9

6 Related Work

• Earlier gating methods route or mix transformed branches while leaving the activation intact or operating at a different point in the layer.Highway Networks, SwiGLU, Mixture of Experts, and Maxout differ in how they gate; Maxout lacks an algebraic inverse to a pretrained weight matrix.Source: §6 · p. 9
• Activation redesign methods alter activation functions within the conventional linear-then-nonlinear paradigm rather than reframing activations as gating.The passage contrasts APL, PAU, KAN, and Curvature Tuning with TG.Source: §6 · p. 9
• TG presents ReLU, SiLU, GELU, Sigmoid, and Tanh as instances of one gating primitive and uses gating to explain their trainability.The passage claims no prior work used gating as the fundamental primitive for neural nonlinearity.Source: §6 · p. 9

7 Limitations and Discussion

• TG kernels are not hand-optimized, so wall-clock training may be slightly slower even though Triton kernels give times comparable to baselines.The speedup claim concerns training steps; width/depth compression varies by model and has room for further research.Source: §7 · p. 9
• The circuit contribution is a reference topology, while detailed implementation and crossbar integration remain future work and ADC/DAC may still be needed for system integration.The passage says TG removes one major obstacle, not that it eliminates all converter needs.Source: §7 · p. 9
• The work targets activations used in general deep-network training, while specialized activations with many inflection points are confined to narrow use cases.Source: §7 · p. 9

8 Conclusion and Future Directions

• TG makes activation choice a continuous per-neuron parameter selection within one primitive, with implicit activations fused into branch-weight gating and the unification extending to softmax attention.The passage attributes direct energy and area benefits to the shift and also cites trainability, training efficiency, compression, and interpretability advantages.Source: §8 · p. 9
• Analog deployment still requires system integration, device-variation handling, and exact circuit implementation; digital hardware could specialize to a parameterized Sigmoid evaluator.The passage also proposes using TG parameter distributions across training stages and layers to explain models and improve training recipes.Source: §8 · p. 9

A Almost Everywhere Equality

• As τ grows, σ(τ(x−θ)) converges to 1[x > θ] at every x except θ, so the functions agree almost everywhere.The disagreement at the single threshold has measure zero and contributes nothing to the summations or integrals used in the work.Source: A Almost Everywhere Equality · p. 11

B Curvature Concentration at the Inflection

• For a well-behaved activation under Gaussian preactivations, the appendix derives an approximation for expected squared curvature in terms of κ(f) and q.The passage attributes the Gaussian preactivation setting to Hayou et al.’s edge-of-chaos analysis.Source: B Curvature Concentration at the… · p. 11
• The approximation assumes f′′ has its dominant magnitude near θ_f and decays in both tails, making secondary sign changes far from θ_f negligible under the Gaussian preactivation.Source: B Curvature Concentration at the… · p. 11
• The approximation is tight when most Gaussian mass lies in the neighborhood where the cubic expansion holds; the cited activations’ neighborhood contains [−3√q, 3√q] at practical depths.The passage links the local property κ at the inflection to the global quantity E[φ′′2] at depth.Source: B Curvature Concentration at the… · p. 12
• In the edge-of-chaos regime, q approaches q* with depth, making the expectation tend to κ(f)2(q* + θ_f2) and the bound Lmax ∝ 1/κ(f)2 activation-specific.Source: B Curvature Concentration at the… · p. 12

• For squared-loss training, the analysis uses the Gauss–Newton approximation and drops the small residual term near a well-fit point, equivalent to using the empirical Fisher information matrix.Source: C Hessian Sensitivity at the Inflection · p. 12
• At the activation inflection, the Hessian is locally flat to first order because its derivative along the pre-activation direction evaluates to zero.Source: C Hessian Sensitivity at the Inflection · p. 12
• At second order, the Hessian curves at a rate proportional to activation complexity κ(f).Source: C Hessian Sensitivity at the Inflection · p. 12
• The bound η2 ≤ δ |f′(θf)|/κ(f), or η = O(1/√κ), means large κ forces smaller steps near the inflection and also governs the TG approximation residual and trainable-depth bound of Hayou, Doucet, and Rousseau [6].Source: C Hessian Sensitivity at the Inflection · p. 12

D Training Details

• Section 3 runs used a fixed seed within each experiment and one RTX 4000 Ada; CIFAR scans matched ReLU optimizer and epoch budgets, while AN4 used a DeepSpeech2 model.The ResMLP and ResNet scans used seed 42; the AN4 setup included a 2-layer convolutional front-end and 5-layer bidirectional LSTM.Source: D Training Details · pp. 12–13
• Constant-family LSTM gates, with fixed sigmoid and tanh branches, outperform multiplicative variants at every H; multiplicative variants are no better than standard LSTM at H≥256.Source: D Training Details · p. 13
• 2.04±0.19× (CIFAR-10/SiLU), 1.98±0.35× (CIFAR-100/GELU), 1.52±0.21× (DS2/AN4 standard LSTM), and 1.47±0.08× (MambaMIM/MSD Hippocampus SiLU) were the TGW training-step speedups.Each is a three-seed mean±standard deviation; on step-matched Mamba checkpoints, TGW wins 50/57 comparisons (sign-test p≈4×10^-9).Source: D Training Details · p. 13

E LLM Conversion: Procedure and Per-Component Breakdown

• Pretrained Transformer conversion replaces attention softmax with its TG form and FFN SiLU/GELU with closed-form TG instances, requiring no fine-tuning, calibration data, or optimizer steps.Source: E LLM Conversion: Procedure and… · p. 13
• SmolLM-135M and TinyLlama-1.1B match their softmax baselines to four decimal places, while GPT-2’s full-conversion perplexity drifts +0.22% from the K=2 GELU approximation, not softmax conversion.The component columns isolate the softmax and FFN activation conversions; the LSE-θ softmax conversion is exact.Source: E LLM Conversion: Procedure and… · pp. 13–14, Table 6 · p. 14
• TG’s τ matches the lowest nontrivial Taylor term at θf; for GELU, the paper adopts τ=1.702 for compatibility instead of the derived τ≈1.596, with per-layer calibration absorbing the residual.Source: E LLM Conversion: Procedure and… · p. 14
• K=3 TG gates are normalized product-form soft indicators over threshold regions, and ViT-B/16 GELU parameters were RMS-fit on sampled pre-activations from 640 ImageNet validation images.The accuracy-test images were disjoint from the calibration set.Source: E LLM Conversion: Procedure and… · p. 14