PaperlayerView exact revision on arXiv

Source-linked AI summary

Rationally Linearized MLPs

Arijit Sehanobish, Byeongchan Kim, Avinava Dubey, Min-hwan Oh, Krzysztof Choromanski

arXiv:2610.13568v1cs.LG

TL;DR

MLP linearization faces an accuracy challenge because accurate, low-variance approximations are difficult. RaLiMP uses low-degree rational activations in a separable feature construction, and experiments report efficient linearization with minimal quality loss across varied tasks.

  • Problem

    Linearized MLPs can have an accuracy gap because highly accurate, low-variance approximations are difficult to obtain.

    Source: §1 · p. 2

  • Method

    RaLiMP linearizes low-degree rational activation surrogates using separable feature blocks for polynomial and rational components.

    Source: §1 · p. 2, §2.5 · p. 6

  • Results

    Across implicit 3D modeling, visual representation learning, semantic segmentation, and language modeling, RaLiMP improves MLP computational footprint with minimal quality loss.

    Source: §4 · p. 10

  • Takeaways & Limitations

    RaLiMP provides an efficient way to replace MLPs in deep networks while retaining quality across the evaluated tasks.

    Source: §2.5 · p. 6, §4 · p. 10

Abstract

from arXiv · show

We present Rationally Linearized Multilayer Perceptrons(RaLiMPs), a new approach to efficiently linearizing MLPs using low-degree rational approximations of standard activation functions. Inspired by polynomial-based MLP linearization, we show that rational approximations can provide accurate representations at substantially lower degree. RaLiMPs exploit these approximations to construct kernel representations with narrow bottlenecks, enabling efficient composition across MLP layers. We provide theoretical guarantees for the resulting approximation and evaluate RaLiMPs across applications ranging from implicit 3D models to large language models. Our experiments demonstrate favorable accuracy--efficiency tradeoffs, with RaLiMPs achieving competitive or improved accuracy while substantially reducing inference cost.

1 INTRODUCTION AND RELATED WORK

MLPs can become a computational bottleneck, while existing linearized layers face an accuracy gap because accurate, low-variance approximation is difficult. The paper uses low-degree rational activations within the linearized MLP framework and evaluates the resulting method across tasks.

1 INTRODUCTION AND RELATED WORK

  • When d^2 ≫ L, MLPs can become Transformer bottlenecks despite improvements to efficient attention, and MLPs also appear in non-attention models.Their computation is quadratic in input dimensionality, while attention is quadratic in sequence length.Source: §1 · p. 1
  • Linearized layers approximate f(Wx) by Ψ(W)Φ(x), separating input-dependent and weight-dependent features before combining them by a dot product.Prior randomized feature constructions can provide unbiased estimates, but high accuracy with low variance is difficult.Source: §1 · pp. 1–2
  • Low-variance polynomial random features address some activation classes, while other efficiency methods often depart from low-variance estimation of regular MLPs.Structured-weight approaches include Monarch matrices and Block Tensor-Train matrices.Source: §1 · p. 2
  • RaLiMP revisits linearized MLPs with low-degree rational activations, which the authors say are easier to approximate accurately than higher-degree functions and support an attractive computational footprint.The paper presents the layer with theoretical results and evaluates it across 3D modeling, vision, and language modeling.Source: §1 · p. 2

2 RATIONALLY LINEARIZED MULTILAYER PERCEPTRONS (RALIMPS)

RaLiMP replaces an activation with a rational surrogate, decomposes that surrogate into polynomial and irreducible-quadratic parts, and represents the parts using separable features. Its error combines rational-fit, quadrature-truncation, and feature-estimation terms; rational blocks can improve fit without increasing variance with the number of poles.

2.1 PRELIMINARIES

  • The activation is replaced by a rational surrogate, whose polynomial part and irreducible-quadratic atoms each receive separable feature blocks that concatenate into one input–weight inner product.The resulting layer computes y = Ψ(W)Φ(x).Source: §2.1 · pp. 2–3

2.2 HANDLING POLYNOMIAL PARTS

  • The polynomial part uses EUGen product features of width 1 + Km, though any finite-variance separable mechanism may be used.The block approximates the polynomial component P(t).Source: §2.2 · p. 3

2.3 HANDLING IRREDUCIBLE-QUADRATIC ATOMS: CENTERED ATOM 1 1+t2

  • The centered atom 1/(1 + t^2) is represented by a truncated Gauss–Laguerre frequency quadrature followed by separable features for each cosine term.Dropping nodes above smax bounds feature growth at the cost of discarding part of the integral.Source: §2.3 · p. 3

2.3.1 THE RATIONAL RANDOM FEATURES

  • Gaussian softmax-kernel features provide unbiased cosine and sine estimates; for the centered atom, the cosine features are concatenated into length-2m towers per frequency.The input-side tower carries quadrature weights and signs, while the weight-side tower depends only on the weight.Source: §2.3.1 · pp. 3–4

2.3.2 VARIANCE ANALYSIS

  • Estimator variance depends on input norms and grows exponentially with the frequency scale, so cutoff and layer scale control it; for summed atoms, the narrowest pole width also governs variance.The scale τ is calibrated per layer as the practical handle on the exponent.Source: §2.3.2 · pp. 4–5

2.4 HANDLING IRREDUCIBLE-QUADRATIC ATOMS: GENERAL SHIFTED ATOM

  • Shifted conjugate-pole atoms reuse the centered atom’s estimators at rescaled frequencies, with the shift mixing cosine and sine estimates; J atoms yield width F = 1 + Km + 4Jnfreqm.The centered atom is recovered when p = 0 and q = 1.Source: §2.4 · p. 5

2.5 APPROXIMATION ERROR GUARANTEES

  • At equal polynomial degree, rational atoms cut bias by one to two orders of magnitude on standard activations and remain bounded where a polynomial diverges.The layer combines polynomial features, quadrature over Laplace representations of rational atoms, and random features for unbiased softmax-kernel estimation.Source: §2.5 · pp. 5–6

3 EXPERIMENTS

The experiments test feed-forward-layer approximation, Transformer replacements across modalities, and generalization beyond Transformers, comparing RaLiMPs with efficient alternatives and dense baselines.

3.1 SYNTHETIC EXPERIMENTS

  • On synthetic feed-forward-layer tests, RaLiMP achieves significantly lower MSE than low-rank, SNNK, and EUGen across all tested activations.The comparison measures MSE between the fully connected layer output and each method’s prediction.Source: §3.1 · p. 6

3.2 RALIMPS IN TRANSFORMERS

  • RaLiMP outperforms other efficient MLP substitutes on classification benchmarks and leads across VTAB datasets and configurations, with inference parameters reduced by up to 57%.VTAB tasks provide only 1,000 training examples each.Source: §3.2 · pp. 6–7, Figure 3 · p. 7
  • On GPT-2 Large/OpenWebText, RaLiMP achieves the best validation perplexity among efficient methods while improving throughput and reducing memory by up to 50%.The perplexity comparison uses matched inference parameters.Source: §3.2 · pp. 7–8, Figure 4 · p. 8
  • At k=8, Qwen3-8B replacement removes 1.17B parameters (∼14%) and raises decoding throughput by 6%, while benchmark accuracy stays near-parity with the dense model.MATH500 and GSM8K slightly exceed the dense model; AIME pass@16 remains close.Source: Table 1 · p. 7, §3.2 · p. 8
  • On GLUE with DistilBERT, RaLiMP is competitive with the baseline while reducing inference parameters by 35%.Source: §3.2 · p. 8
  • Across ScanNet, ScanNet200, and nuScenes, RaLiMP beats competing efficient methods and remains near the dense baseline even when all 22 PTv3 MLP modules are replaced.At k=22, the ScanNet and ScanNet200 gaps are 0.24 and 0.40 mIoU points; nuScenes is slightly above baseline.Source: Table 2 · p. 8, §3.2 · p. 9
  • Compared with the dense PTv3 baseline, RaLiMP reduces memory by up to 20% and makes MLPs up to 2× faster; k=10 captures most gains.Replacing the remaining 12 modules saves only 2.8 MB and 0.08 ms, while reducing segmentation accuracy across benchmarks.Source: §3.2 · p. 9, Table 3 · p. 9

3.3 BEYOND TRANSFORMERS

  • Beyond Transformers, RaLiMP reduces model size by 30%–50% while matching or improving PiNN, NeRF, and SIREN performance, and closes the dense-model gap on Burgers-equation PINNs.For NeRF, inference time falls by 23%; the PINN model is 42% smaller.Source: Figure 5 · p. 9, §3.3 · pp. 9–10

4 CONCLUSION

The conclusion presents RaLiMPs as efficient linearized MLPs evaluated across varied tasks, while the appendices establish unbiased rational-feature estimates and characterize their variance. The variance for sums of rational atoms is governed by the narrowest pole width.

4 CONCLUSION

  • RaLiMPs are presented as efficient linearized MLPs that approximate common feed-forward layers, with experiments spanning implicit 3D modeling, visual representation learning, semantic segmentation, and language modeling.The authors report improved computational footprint with minimal quality loss.Source: §4 · p. 10

A.1 UNBIASEDNESS OF THE RATIONAL RANDOM FEATURES

  • For general τ, the rational random-feature estimates are unbiased for cos(st/τ^2) and sin(st/τ^2).The τ = 1 case is established first, then extended by rescaling the inputs.Source: §A.1 · p. 20

A.2 VARIANCE OF THE RATIONAL RANDOM FEATURES

  • For a sum of rational atoms, the variance is governed by the narrowest pole width qmin, rather than by smax and τ alone.The variance depends on the inputs through ∥x + w∥ and grows exponentially in 2σ^2.Source: §A.2 · p. 21

A.3 TOTAL APPROXIMATION ERROR

The total approximation error on a bounded interval combines rational-fit error, quadrature truncation error, and random projection variance. These terms have distinct controls, while the input scale links variance to the quadrature cutoff.

A.3 TOTAL APPROXIMATION ERROR

  • On an interval containing the pre-activations, the total error combines rational-fit error, quadrature truncation error, and variance from finite random projections.The estimator is unbiased for the truncated surrogate; the rational fit and truncated quadrature contribute deterministic error.Source: §A.3 · p. 21
  • More poles reduce rational-fit error, while more quadrature nodes and a larger cutoff reduce quadrature error; more projections reduce variance.Raising smax also inflates rational-feature variance exponentially, so the cutoff and input scale cannot be varied independently.Source: §A.3 · p. 22
  • Hoeffding- and Bernstein-type concentration bounds require bounded projections because the estimator summands have lognormal tails and an infinite moment-generating function.Under bounded projections, the concentration bound has an explicit failure probability, with a constant governed by the same exponential as the variance.Source: §A.3 · p. 22

A.4 COST OF ACCURACY

The appendix contrasts the accuracy cost of rational and polynomial feature blocks, describes how RaLiMP constructs its kernel representation, and explains how input scaling controls variance and effective frequencies.

A.4 COST OF ACCURACY

  • For SiLU on [−T, T], the polynomial product-feature width scales as ε^−2−α, while a rational block approximates an atom to error ε with O_q(ε^−2).The rational block’s cost depends on pole width q as a constant, whereas the polynomial cost has a growing exponent in ε.Source: §A.4 · p. 23

B UNIFIED KERNEL REPRESENTATION

  • The rational block uses a nonnegative exponential spectral measure for each irreducible-quadratic atom, with Gauss–Laguerre quadrature and learnable pole locations and widths.The continuous spectral density must be discretized into frequencies; pole width rescales the nodes and weights.Source: B UNIFIED KERNEL REPRESENTATION · p. 23, Figure 6 · p. 24

C.1 CHOOSING THE INPUT SCALE τ

  • The scale τ controls both the variance exponent and the effective frequencies, so larger τ reduces variance but compresses quadrature nodes toward the atom’s flat center.The useful regime is t/τ^2 = O(1), placing nodes where the atom varies.Source: §C.1 · p. 24

C.2 DETERMINISTIC NODES VERSUS MONTE CARLO

The appendix explains why deterministic quadrature nodes are preferred to Monte Carlo nodes and extends the construction to repeated poles and efficient layer composition. It also describes experiments and a synthetic benchmark used to evaluate the method.

C.2 DETERMINISTIC NODES VERSUS MONTE CARLO

  • Deterministic Gauss–Laguerre nodes outperform Monte Carlo nodes at every tested count, with a 7.4 × 10−3 versus 1.2 × 10−2 error at nfreq = 3 and a two-orders-of-magnitude gap at nfreq = 8.Least-squares-optimal trainable coefficients are used for both node sets; deterministic nodes also make the discarded-weight bound deterministic, with randomness confined to projections.Source: §C.2 · p. 24

C.3 REPEATED CONJUGATE POLES

  • Repeated conjugate poles need no new feature maps: polynomial Laplace weights on cosine and sine channels preserve the existing kernel, estimators, and variance bound.The weights have degree n−1 for a pole of multiplicity n.Source: §C.3 · p. 25

C.4 COMPRESSING NETWORKS BY COMBINING LAYERS

  • A following fully connected linear layer can be folded into the RaLiMP projection, allowing the experiments’ consecutive layers to be collapsed for more efficient inference.The folded weight is Ŵ = W′Ψ(W), with bias b′ retained.Source: §C.4 · p. 25

C.5 EFFICIENT COMPUTATIONS WITH HADAMARD-PRODUCT MATRICES

  • An unbiased rank-r factorization of the Hadamard-product matrix enables efficient matrix-vector multiplication without materializing d²-length vectors.For LLM gated feed-forward blocks, the weight-side product folds offline, leaving one feature map and two O(rd) matrix products per token.Source: §C.5 · pp. 25–26

D EXPERIMENTAL DETAILS

  • Experiments use Gauss–Laguerre frequencies and retain nodes up to smax when specified; otherwise, all nfreq nodes are kept.A fixed smax retains lower frequencies as nfreq grows; pretrained-backbone input scale is τ = c RMS(w·x), with c = 4 unless stated otherwise.Source: D EXPERIMENTAL DETAILS · p. 26

D.1 SYNTHETIC EXPERIMENTS

  • The synthetic benchmark fits one standardized 256-by-256 layer across five activations, with all methods using a 32-feature budget and RaLiMP allocating 8 features to degree-2 polynomial terms and 24 to rational features.Training uses 10 seeds and reports train MSE; RaLiMP’s companion panel sweeps its feature budget.Source: §D.1 · p. 26

D.2 VIT EXPERIMENTS

The experiments replace selected MLP blocks in ViT and GPT-2 with RaLiMPs. The supplied passages describe their evaluation setups and report reduced memory overhead and faster inference as more ViT layers are replaced.

D.2 VIT EXPERIMENTS

  • Replacing more ViT layers with RaLiMPs reduces memory overhead and speeds inference.The experiments use ViT-B/16 across CIFAR-100, Places365, ImageNet-1k, and 19 VTAB-1k transfer tasks, replacing the MLPs in k = 4, 7, 9, or 11 blocks.Source: §D.2 · p. 26, Figure 7 · p. 27

D.3 CAUSAL LANGUAGE MODELING

  • GPT-2 Large is trained from scratch on OpenWebText, with the MLPs in the last k blocks replaced and validation perplexity reported after 30,000 iterations.RaLiMP and EUGen use degree-2 polynomials; all methods use 128 random features, with RaLiMP splitting them between polynomial and rational blocks.Source: §D.3 · pp. 27–28

D.4 PTV3 EXPERIMENTS

The appendix details RaLiMP experiments in point-cloud segmentation and coordinate networks, including SIREN, NeRF, and physics-informed models. In the reported SIREN and Burgers’ equation comparisons, RaLiMP matches or reproduces key features of the reference at the stated inference budget.

D.4 PTV3 EXPERIMENTS

  • PTv3 experiments compare EUGen, SNNK, and RaLiMP on indoor ScanNet and ScanNet200 and outdoor nuScenes segmentation, matching the number and locations of replaced MLP modules.The reported metrics are mIoU, mean class accuracy, and overall accuracy.Source: §D.4 · p. 28, Table 6 · p. 28

D.5 COORDINATE NETWORKS

  • The coordinate-network experiments cover NeRF, SIREN, and physics-informed networks.The section introduces the detailed experimental settings for these network types.Source: §D.5 · p. 30

D.5.1 SIREN

  • RaLiMP matches the dense SIREN baseline while replacing all hidden layers and outperforms all tested variants.The image-fitting comparison uses matched inference parameters for the linearized methods.Source: Figure 8 · p. 30, §D.5.1 · p. 30

D.5.2 NERF

  • The NeRF experiment replaces feed-forward layers in eight synthetic Blender scenes and reports mean PSNR over test views.All linear methods use matched 256 random features.Source: §D.5.2 · p. 30

D.5.3 PHYSICS-INFORMED NETWORK

  • At matched inference budget, RaLiMP and the dense tanh network sharply reproduce the Burgers’ shock, while EUGen and SNNK smear it.The comparison replaces all hidden layers; the RaLiMP and competing linearized variants use fewer deployed parameters than the tanh network.Source: Figure 9 · p. 31, §D.5.3 · p. 31

D.6 QWEN3-8B EXPERIMENTAL DETAILS

  • The Qwen experiments are described in a separate appendix section.The supplied passage introduces experimental details but gives no Qwen outcomes.Source: §D.6 · p. 31

D.6.1 DATASETS

The Qwen3-8B experiments replace feed-forward layers in the top four or eight blocks and use supervised fine-tuning followed by on-policy distillation. The reported distillation gains are largest on longer-generation benchmarks and recover most of the supervised model’s accuracy loss.

D.6.1 DATASETS

  • OpenThoughts3-1.2M supplies supervised fine-tuning traces, while DeepScaleR supplies competition-math prompts for on-policy rollouts; the retained distillation set also includes science traces.The k = 8 retained set contains 87,000 math pairs; the k = 4 set contains 20,000, and the science pool has 2,287 traces.Source: §D.6.1 · pp. 31–32
  • The replacement uses a rational-feature SiLU gate and collapses the gated product to a rank-r factorization; k = 8 and k = 4 replace the top eight and four blocks, respectively.The two-stage recovery pipeline uses supervised fine-tuning followed by on-policy distillation; the k = 4 model requires less data in both stages.Source: §D.6.1 · p. 32

D.6.2 STAGE 1: SUPERVISED FINE-TUNING OF THE REPLACEMENT LAYERS

  • Stage 1 fits freshly initialized replacement layers to the base model on OpenThoughts3-1.2M, selecting a checkpoint by held-out OpenThoughts validation accuracy.The resulting compressed model is the pre-distillation model.Source: §D.6.2 · p. 33

D.6.3 STAGE 2: SAMPLING THE COMPRESSED MODEL’S OWN SOLUTIONS

  • Stage 2 samples eight solutions per problem from the compressed model and keeps only self-terminating solutions whose final boxed answers are correct.These retained model-generated solutions form the on-policy distillation set.Source: §D.6.3 · p. 34

D.6.4 STAGE 3: ON-POLICY DISTILLATION

  • Stage 3 trains the compressed student against the frozen original teacher using forward KL on the student’s own generated solutions.This scores the teacher on states the compressed model actually visits.Source: §D.6.4 · p. 34

D.6.5 EVALUATION PROTOCOL

  • Small AMC, AIME, and GPQA datasets use 16 samples per problem for avg@16 and pass@16; larger benchmarks use one sample per problem.Mathematical answers are extracted from the final boxed answer and checked against ground truth.Source: §D.6.5 · pp. 34–35

D.6.6 EFFECT OF THE DISTILLATION STAGE

  • After on-policy distillation, GSM8K and MATH500 exceed uncompressed Qwen3-8B, while Minerva and OlympiadBench come within a point; the largest gains occur on the longest-generation benchmarks.For k = 8, the supervised model’s accuracy loss grows with base generation length, and distillation recovers most of it.Source: Table 9 · p. 35, §D.6.6 · p. 35

D.6.7 LENGTH-COMPOUNDING OF THE KERNEL APPROXIMATION ERROR

Kernel approximation error compounds with generation length, while increasing Hadamard rank reduces factorization error and improves competition-math results. Replacing feed-forward networks reduces model parameters, with greater savings when more blocks are replaced.

D.6.7 LENGTH-COMPOUNDING OF THE KERNEL APPROXIMATION ERROR

  • 3% at L≈1.5k rises to 41% at L≈16k; the deficit compounds with generation length, and δ grows with swapped-layer count but shrinks with feature rank.The implied rate is ∼2–4×10−5 per token where the base model is unsaturated; distillation preferentially removes this length-dependent error.Source: §D.6.7 · p. 35, Figure 10 · p. 36

D.6.8 EFFECT OF THE HADAMARD RANK r

  • Raising Hadamard rank from r=256 to r=512 lowers factorization error and improves competition-math benchmarks for the k=4 variants.The main-body table reports the r=512 variant.Source: §D.6.8 · p. 35, Table 10 · p. 36

D.6.9 MEMORY FOOTPRINT AND THROUGHPUT

  • At k=8, replacement removes 1.17B parameters; at k=4, it removes 0.59B from the 8.19B-parameter model.Savings grow linearly with replaced blocks; the feed-forward networks account for about two-thirds of model parameters.Source: §D.6.9 · p. 36, Table 11 · p. 36

E DETAILED RESULTS

The detailed results cover NeRF, ViT, VTAB-1k, semantic segmentation, and causal language modeling. Captions report wins at matched inference budgets for NeRF and ViT, and improved GPT-2 Large perplexity over two baselines at every replacement depth.

NeRF

  • RaLiMP beats all other methods on held-out Synthetic-NeRF PSNR at matched deployed parameters, including the dense NeRF baseline.The comparison is reported in dB.Source: Table 12 · p. 37

ViT

  • RaLiMPs beat all other methods on ViT-B/16 Top-1 accuracy at matched deployed inference budget across CIFAR-100, ImageNet, and Places365.The baseline is the dense ViT-B/16 full fine-tune.Source: Table 13 · p. 37

VTAB-1k

  • The VTAB-1k table reports top-1 accuracy at matched inference budget for each number of replaced layers, comparing RaLiMP, SNNK, EUGen, and Lowrank.Bold marks the best method for each task and replacement depth.Source: Table 14 · p. 37

Semantic segmentation and causal language modeling

  • The ScanNet v2 and nuScenes tables report validation mIoU, while GPT-2 Large perplexity improves on EUGen and SNNK at every replacement depth.The GPT-2 Large comparison is on OpenWebText validation at a fixed 30k-iteration budget; lower perplexity is better.Source: Table 17 · p. 38, Table 15 · p. 38, Table 16 · p. 38

F ADDITIONAL EXPERIMENTS

RaLiMP further compresses DistilBERT while remaining competitive on GLUE. The related-work discussion positions its extension to rational-function kernels alongside prior linearization and efficiency approaches.

F ADDITIONAL EXPERIMENTS

G ADDITIONAL RELATED WORK

  • Prior random-feature kernel approaches are restricted to positive definite kernels, with Kim et al. (2025) as a recent exception; this work extends linearization to special classes of rational functions.The discussion also situates the work among JLT and random-feature methods for reducing neural-network time and space complexity.Source: G ADDITIONAL RELATED WORK · p. 39
  • NTK analyses provide training-dynamics insights but do not directly reduce from-scratch MLP computation; URF separates weights and inputs but relies on activation Fourier transforms.Kim et al. (2025) uses random features for polynomial activations and polynomial approximation on bounded intervals for continuous activations.Source: G ADDITIONAL RELATED WORK · p. 39
  • The related-work discussion also lists pruning, quantization, distillation, and structured matrices for dense-layer efficiency, alongside NeRF-specific methods that do not generalize beyond that domain.The cited methods are pruning, quantization, and distillation (Li et al., 2023; Sreenivas et al., 2024; Busbridge et al., 2025).Source: G ADDITIONAL RELATED WORK · pp. 39–40
Loading 2610.13568v1…