PaperlayerView exact revision on arXiv

Source-linked AI summary

SPA: Structure-Preserving Approximation of Gated Activations under FHE

Rui Li, Wenyuan Wu, Weijie Miao

arXiv:2610.32065v1cs.CR

TL;DR

FHE does not directly implement nonlinear activations such as GELU and SiLU, creating a challenge for encrypted neural inference. SPA preserves the known odd component and approximates the even remainder; fixed-interval CKKS benchmarks report favorable error-latency trade-offs, including a speedup for erf-GELU and lower RMSE with a speedup for QuickGELU.

  • Problem

    FHE enables neural inference on encrypted inputs but does not directly implement nonlinear activations such as GELU and SiLU.

    Source: §1 · p. 1

  • Method

    SPA retains the known odd part of structured activations and approximates the even remainder after changing variables to u=x^2.

    Source: §1 · p. 1

  • Results

    On R5, SPA-balanced is 3.25× faster than MOAI for erf-GELU; SPA-accurate gives 79.22% lower QuickGELU RMSE and 10.40× speedup over Euston on [−5,5].

    Source: §5.4 · pp. 7–8

  • Takeaways & Limitations

    The restriction reduces free coefficients without lowering best uniform approximation error at the corresponding degree.

    Source: §3.1 · p. 3

    Model results are empirical, with task-specific calibration in two LLMs, and kernel speedups do not establish end-to-end gains.

    Source: §6 · p. 9

Abstract

from arXiv · show

Nonlinear activation evaluation remains a major bottleneck in neural inference under fully homomorphic encryption (FHE). FHE supports efficient addition and multiplication, but lacks native operations for exponential functions, reciprocals, and value-dependent selection. High-degree polynomial approximations on fixed intervals are therefore commonly used to replace nonlinear activations. They can achieve small errors but often incur substantial multiplicative depth and latency. We observe that several widely used nonlinear activations share the algebraic structure $A(x)=xg(x)$ with $g(-x)=1-g(x)$. This class includes GELU, SiLU, fixed-parameter Swish, and the scalar nonlinear branches of GEGLU/SwiGLU. We propose \SPA{} (\emph{Structure-Preserving Approximation}) to exploit this shared structure. \SPA{} retains the exact odd component $x/2$ and approximates only the even residual as a polynomial in $x^2$, yielding a comparison-free arithmetic kernel. This construction removes redundant odd degrees of freedom and enables lower-cost polynomial evaluation at a given error target. Plaintext experiments with Gemma-4-E2B and Llama-3.2-3B show that SPA largely preserves downstream task performance without weight updates. Fixed-interval CKKS benchmarks show favorable error--latency trade-offs. On $[-5,5]$, \SPA{}-balanced achieves a $3.25\times$ speedup over MOAI (ICLR'~26) for erf-GELU with comparable error. On the same interval, \SPA{}-accurate achieves $79.22\%$ lower root mean squared error (RMSE) and a $10.40\times$ speedup over Euston (S\&P'~26) for QuickGELU. For SiLU, \SPA{}-accurate achieves $19.02\%$ lower RMSE and a $2.06\times$ speedup over THOR (CCS'~25).

1 INTRODUCTION

FHE makes nonlinear activations costly to evaluate, while GELU, SiLU, fixed-parameter Swish, and the scalar branches in GEGLU/SwiGLU share a useful symmetry. SPA preserves the exact odd component and approximates only the even remainder, with the introduction also describing its approximation and evaluation contributions.

1 INTRODUCTION

  • FHE lacks direct implementation of nonlinear activations, making accurate, low-cost replacements important for balancing polynomial accuracy and depth against lookup, selection, or communication costs.The paper studies scalar branches; replacing one leaves the other projection, elementwise product, and trained weights unchanged.Source: §1 · p. 1
  • GELU, SiLU, fixed-parameter Swish, and the scalar nonlinear branches of GEGLU/SwiGLU have the structure A(x)=xg(x), g(-x)=1-g(x).This implies that A(x)-A(-x)=x, so the odd part is known exactly and only the even remainder requires approximation.Source: §1 · p. 1, Figure 1 · p. 2
  • SPA retains the exact odd part and approximates the even remainder as a polynomial in x^2, using the shared activation structure.The method uses Chebyshev polynomials and discrete minimax fitting, with an explicit evaluation schedule and coefficient-specific error bounds.Source: §1 · pp. 1–2
  • SPA's structural restriction preserves the best uniform approximation error at the corresponding degree in x, independently of the basis.The introduction also notes that the equivalent even-polynomial representation has the same approximation space.Source: §1 · p. 2

2 RELATED WORK

Prior secure-activation methods use different mechanisms and incur different costs, while SPA and MOAI differ in their approximation targets and evaluation schedules. The cited system-level timings are not directly comparable with scalar-kernel timings.

2 RELATED WORK

  • Degree, interval, and evaluation schedule affect homomorphic cost, while lookup, piecewise, MPC, and hybrid approaches can incur access, selection, or communication costs.Range calibration reduces Sylph's SiLU approximation degree from 83 to 31.Source: §2 · p. 2
  • MOAI approximates 2g−1 and reconstructs x[1+bf(x)]/2, whereas SPA directly approximates the even residual x[g(x)−1/2] as q(x^2).The methods differ in their approximation targets and evaluation schedules.Source: §2 · p. 2
  • Polynomial Transformer and Powerformer combine homomorphic activation strategies with broader model or operator techniques, so their system timings cannot be compared directly with scalar-kernel timings.Powerformer's GELU remains an untested baseline; OLA and ATLAS address model-level allocation (Lee et al., 2023; Xie et al., 2026).Source: §2 · pp. 2–3

3 STRUCTURE-PRESERVING POLYNOMIAL APPROXIMATION

SPA retains the activation’s exact odd component and approximates its even residual as a polynomial in x^2. The section establishes the approximation-space result, describes target fitting, and gives an encrypted evaluation schedule.

3.1 ACTIVATION STRUCTURE AND CHANGE OF VARIABLE

  • The even residual is represented as H_A(x^2), so SPA retains the linear term while approximating only the residual.The implementation maps x^2 to a normalized variable and uses a Chebyshev representation that reduces free coefficients, not degree in x.Source: §3.1 · p. 3, Figure 2 · p. 4
  • For every integer k ≥ 1, symmetry preserves the best uniform approximation error for the structured family, with the uniform norm defined on the stated compact intervals.The theorem characterizes attainable error but does not guarantee numerical fitting attains it.Source: §3.1 · p. 3

3.2 TARGETS AND COEFFICIENT CONSTRUCTION

  • The main experiments target SiLU, erf-GELU, and QuickGELU.The coefficient-construction procedure fits the transformed residual on a finite node set using an unweighted discrete minimax program, retaining the first degree meeting the selected grid tolerance.Source: §3.2 · p. 4

3.3 HOMOMORPHIC POLYNOMIAL EVALUATION

  • Polynomial division produces blocks for a Paterson–Stockmeyer representation, which encrypted evaluation combines using powers of the transformed input.Coefficient transformations are performed offline; encrypted evaluation requires no comparison or transcendental operation.Source: §3.3 · p. 4
  • The ciphertext–ciphertext multiplication count depends on the chosen split between block size and block count.The reported splits are fixed rather than runtime-optimized, and pruning can reduce this schedule-specific count.Source: §3.3 · p. 5

4 ERROR BOUNDS FOR SPECIFIED COEFFICIENTS

Section 4 turns best-approximation results into error bounds for specified coefficients and evaluated outputs. It distinguishes interval-wide guarantees from sampled checks and coefficient- or evaluation-specific error.

4.1 UNIFORM ERROR AND COEFFICIENT PERTURBATIONS

  • A finite covering bounds interval error using the grid error and a derivative bound; every input must lie within h of an evaluated point.The mean value theorem gives the between-point contribution, while the derivative bound can be conservative.Source: §4.1 · p. 5
  • For the same radius, Chebyshev basis, and fixed linear term, coefficient differences yield a transfer bound through the coefficient-vector distance.The bound uses |Tj(t)| ≤ 1 and is basis-dependent.Source: §4.1 · p. 5
  • A numerical certificate requires outward-rounded enclosures for the grid errors, derivative bound, and arithmetic; directly enclosing the error derivative may tighten the conservative bound.The stated derivative bound uses MA ≥ ∥A′∥∞.Source: §4.1 · p. 5

4.2 APPROXIMATION AND EVALUATION ERROR

  • The decoded-output error is bounded by fitting/reference mismatch, approximation error, coefficient perturbation, and homomorphic evaluation error.For the main ciphertext comparisons, the fitting/reference mismatch term vanishes; the final term includes numerical evaluation and input representation errors.Source: §4.2 · p. 5
  • Rounding k + 1 coefficients to ℓ decimal places bounds their l1 perturbation by (k + 1)10^-ℓ/2; binary storage error is additional.An upper bound above budget is inconclusive, and measured finite-grid discrepancies are not certified bounds over all ciphertexts.Source: §4.2 · p. 6

5 EVALUATION

The evaluation tests frozen-model quality and ciphertext error–latency trade-offs across activations and intervals. SPA-balanced largely preserves pooled accuracy, while SPA variants report favorable per-activation comparisons alongside sampled precision checks.

5.1 EXPERIMENTAL SETUP

  • The frozen-weight tests use Gemma-4-E2B-it and Llama-3.2-3B-Instruct; ciphertext kernels are evaluated separately from plaintext model evaluations.MCQ and language-modeling tasks are evaluated with specified calibration and scoring procedures.Source: §5.1 · p. 6

5.2 BASELINES AND ENCRYPTED OPERATION COUNTS

  • Baselines vary by activation, and Table 2 reports per-call operation counts, including input-mapping costs; methods generally share a CKKS context within each target and interval.QuickGELU SPA uses a 46-bit scale, while Euston uses a separate 35-bit context; no kernel uses bootstrapping.Source: §5.2 · pp. 6–7, Table 2 · p. 6

5.3 FROZEN-MODEL QUALITY

  • SPA-balanced changes pooled accuracy by −0.04 percentage points for Gemma and +0.09 for Llama; Gemma PPL rises 0.17%/0.21% versus MOAI’s 2.88%/2.47% on WikiText-2/C4-en.The Gemma pooled-accuracy paired 95% interval is [−0.1422, +0.0711].Source: §5.3 · p. 7, Table 3 · p. 7, Figure 3 · p. 7
  • NEXUS produces 1,609 invalid MCQ predictions and fails on 103/368 WikiText-2 and 534/1,501 C4-en windows, so shared-window gains do not establish full-corpus gains.Source: §5.3 · p. 7

5.4 CIPHERTEXT ACCURACY AND LATENCY

  • For erf-GELU, SPA-balanced is 3.25× faster than MOAI on R5; for SiLU, SPA-accurate has 19.02% lower RMSE and a 2.06× speedup over THOR.On erf-GELU R20, SPA-balanced has 76.23% lower RMSE and a 1.80× speedup over MOAI.Source: §5.4 · pp. 7–8, Figure 4 · p. 8
  • For QuickGELU on [−5, 5], SPA-accurate achieves 79.22% lower RMSE and a 10.40× speedup over Euston.The ciphertext comparison uses a common grid; Euston uses a different CKKS context from SPA.Source: §5.4 · p. 8, Figure 4 · p. 8, Table 4 · p. 9

5.5 COEFFICIENT AND CKKS PRECISION

  • Three-decimal coefficient rounding exceeds 10^-3 maximum error; four decimals meet that budget for erf-GELU and QuickGELU, while SiLU reaches 1.01 × 10^-3.Unrounded coefficients meet the budget at 46/42-bit CKKS precision but not 38 bits; these are sampled observations, not uniform guarantees.Source: §5.5 · p. 8, Figure 5 · p. 9

6 LIMITATIONS AND CONCLUSION

SPA preserves the known odd part and best uniform approximation error for activations with the specified structure, while the reported empirical and scalar guarantees have defined limits.

6 LIMITATIONS AND CONCLUSION

  • SPA preserves the known odd part and best uniform approximation error for activations with the specified structure.The measured error–cost trade-offs do not establish superiority over all optimized even-polynomial evaluators.Source: §6 · p. 9
  • Model results are empirical, use task-specific calibration in two LLMs, and Gemma uses Exact-erf.Scalar guarantees exclude unverified configurations and out-of-range inputs.Source: §6 · p. 9
  • Kernel speedups do not establish end-to-end gains.Source: §6 · p. 9

REPRODUCIBILITY STATEMENT

The statement describes reproducibility materials, then develops the structural approximation proof, regularity conditions, and activation-specific residual observations.

REPRODUCIBILITY STATEMENT

  • Stored coefficients, original result records, and scripts are provided to regenerate numerical tables and figures.Data processing, calibration, and paired evaluation are described in Appendix C; generative AI assisted with code organization and manuscript wording.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 PROOF OF THEOREM 1

  • The residual rA(x) = A(x) − x/2 is even, so it can be represented through HA(x^2) on the stated interval.The proof argues that every polynomial candidate has a structure-preserving replacement with no greater uniform error.Source: §A.1 · p. 13

A.2 REGULARITY AT THE ORIGIN

  • Theorem 1 assumes continuity, while analyticity of g near zero makes the square-root expression for HA nonsingular at zero for the analytic gates considered.Continuity and parity alone are insufficient for differentiability at zero, as shown by the stated counterexample.Source: §A.2 · p. 13

A.3 RESIDUALS OF THE ACTIVATION EXAMPLES

  • For the analytic construction, summing individual approximants yields bounds on best-approximation errors, not arbitrary stored coefficient vectors.Source: §A.3 · pp. 13–14
  • The Exact-erf Gemma reference and its polynomial replacements use the erf-GELU target, while tanh-GELU pole locations differ from those in the SiLU representation.QuickGELU is fit directly on each stated interval, and SiLU coefficients require scaling before reuse across intervals.Source: §A.3 · p. 14

B.1 PROOF OF THEOREM 2

The appendix presents a finite-grid error-bound argument, discusses a possible local derivative refinement, and records validation status and arithmetic assumptions.

B.1 PROOF OF THEOREM 2

  • A finite covering and derivative bound yield an interval error bound from evaluated points; the coefficient-transfer bound uses the Chebyshev inequality.Source: §B.1 · p. 14
  • For erf-GELU, the stored validation result is a sampled check rather than a certificate because the validator lacks an interval enclosure for erf evaluations.Table 5 distinguishes validated SiLU and QuickGELU bounds from sampled erf-GELU checks; no continuous-interval bound was produced where the table shows a dash.Source: §B.1 · p. 14, Table 5 · p. 15, §B.3 · p. 15

B.2 POSSIBLE REFINEMENT OF THE DERIVATIVE BOUND

  • Direct interval evaluation of the error derivative may reduce overestimation, but a sampled derivative maximum is not a valid substitute for an interval bound.This refinement was not used to generate Table 5.Source: §B.2 · p. 15

B.3 ARITHMETIC ASSUMPTIONS AND CURRENT STATUS

  • The validation implementation relies on stated correct-rounding and range assumptions, and porting requires rechecking them.Independent ball-arithmetic validation has not been performed, and the available results lack a completed coefficient-transfer certificate for the Gemma-erf vectors.Source: §B.3 · p. 15

C PLAINTEXT EVALUATION PROTOCOL

The protocol freezes calibration and range choices before evaluation, specifies zero-shot scoring and paired statistics, and details window and channel-assignment counts.

C PLAINTEXT EVALUATION PROTOCOL

  • The static plan and coefficients are frozen before evaluation, and test labels are not used to choose them.Gemma calibration and range assignment use the Exact-erf model rather than the original tanh trajectory.Source: C PLAINTEXT EVALUATION PROTOCOL · p. 15
  • The main results use zero-shot conditional sequence scores with plain prompts and task-specific input budgets and continuation scoring rules.The protocol is not intended to reproduce public leaderboard results obtained with chat templates.Source: C PLAINTEXT EVALUATION PROTOCOL · p. 15
  • Accuracy differences use paired example-level correctness, with invalid predictions counted as incorrect and 10,000 paired percentile-bootstrap replicates.Pooled accuracy weights examples, while macro accuracy weights tasks equally.Source: C PLAINTEXT EVALUATION PROTOCOL · p. 16
  • Language-modeling scores use shared valid windows and total negative log-likelihood with scored-token counts.Calibration counts in Table 6 are examples or text windows, not independent documents.Source: C PLAINTEXT EVALUATION PROTOCOL · p. 16, Table 6 · p. 16
  • Table 7 reports static range-assignment proportions using all channel-within-layer assignments as the denominator, plus an exact count for wide channels.Wide channels are not layers or tokens, and these proportions are not runtime coverage rates.Source: C PLAINTEXT EVALUATION PROTOCOL · p. 16, Table 7 · p. 16

D CIPHERTEXT SETTINGS AND COMPLETE KERNEL RESULTS

The section reports fixed-interval CKKS kernel settings and CPU timing results, with comparisons across activation targets and approximation variants. The measurements use a specified context and polynomial degree setup, and are not GPU or end-to-end model latencies.

D CIPHERTEXT SETTINGS AND COMPLETE KERNEL RESULTS

E COMPLETE MAIN-TASK RESULTS

This block records complete task-result comparisons, precision diagnostics, and implementation details for the numerical construction and Chebyshev evaluation. It also specifies the comparison references and computational settings used in the reported tables.

E COMPLETE MAIN-TASK RESULTS

  • The main-task tables retain reported methods and failures, compare Gemma against Exact-erf and Llama against Native for MCQ, and use Exact-erf as the Gemma language-modeling reference.Failed MCQ predictions count as incorrect; when language-modeling windows fail, full-corpus perplexity is omitted and paired differences use shared valid windows.Source: Table 11 · p. 18, E COMPLETE MAIN-TASK RESULTS · p. 18, Table 12 · p. 20

F PRECISION DIAGNOSTICS AND COMPACT SUMMARIES

G EXPERIMENTAL ENVIRONMENT AND CODE AVAILABILITY

  • The listed experimental environment uses an Intel Xeon Gold 6530 for CKKS kernels, an NVIDIA RTX PRO 6000 Blackwell Workstation Edition for plaintext models, and SEAL 4.1.The passage also provides the code repository address.Source: G EXPERIMENTAL ENVIRONMENT AND CODE… · p. 21

H.1 RECORDED NUMERICAL CONSTRUCTION

  • The recorded numerical construction solves a finite-node minimax problem with all k + 1 coefficients free, then refines nodes using sampled error on a 300,001-point t-grid.It checks activation error on a separate 300,001-point grid in the original input interval and searches degrees using m = 4 for R5 and m = 8 for R20/wide.Source: §H.1 · p. 21

H.2 CHEBYSHEV DIVISION AND THE PS BLOCKS

  • Chebyshev division converts the input coefficient vector into F = TmQ + B with deg B < m, and repeated application yields the PS blocks.For the recorded dense schedule, forming x^2 and the final product requires five CT×CT operations; normalization, the retained x/2 term, and coefficient products require eight PT×CT operations.Source: §H.2 · pp. 21–22
Loading 2610.32065v1…