PaperlayerView exact revision on arXiv

Source-linked AI summary

Fiona: Accelerating FHE Inference with Packing-Aware Ternary Weights

Yiteng Peng, Zhibo Liu, Dongwei Xiao, Shuai Wang

arXiv:2609.35352v1cs.LG

TL;DR

Packed FHE inference remains costly because plaintext weights require PMult, while ordinary ternarization often fails to avoid those operations and can reduce accuracy. FIONA selectively ternarizes weights by packing group and task sensitivity, then fits lower-degree polynomials under an accuracy budget. It reduces PMult and accelerates encrypted inference on VGG11, ViT, and BERT with less than 1% accuracy loss.

  • Problem

    Packed execution avoids PMult only when all weights in a group share a ternary value, while converting all groups can substantially reduce accuracy.

    Source: §1 · pp. 1–2, §3.1 · p. 5

  • Method

    FIONA uses task sensitivity to selectively ternarize weights within fixed packing groups, retains sensitive groups at full precision, and refits lower-degree polynomials under a cumulative accuracy budget.

    Source: §1 · p. 2, §3.2 · p. 6, §3.3 · p. 7

  • Results

    FIONA reduces #PMult by 53.4–79.5% and accelerates encrypted inference by 2.38× on VGG11, 1.68× on ViT, and 1.84× on BERT, with less than 1% accuracy loss.

    Source: §1 · p. 2, §6.1 · p. 13

  • Takeaways & Limitations

    Experiments on CNN and Transformer models demonstrate speedups with little loss in plaintext accuracy.

    Source: §8 · p. 19

    The threat model assumes an honest-but-curious server, and input shapes and padding masks are public metadata.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

Fully homomorphic encryption (FHE) enables neural network inference directly on encrypted inputs, but it remains orders of magnitude slower than plaintext in- ference. Applying the server's plaintext weights to encrypted activations involves plaintext-ciphertext multiplications (PMult) and accounts for more than half of inference time in recent systems. Ternary quantization can replace these multipli- cations with additions and subtractions, but the savings rarely materialize under packed execution. A single PMult applies a weight group fixed by the packing layout and can be avoided only when all its weights share the same ternary value. Ternarizing all groups, however, largely degrades accuracy. We present FIONA, an offline optimizer that selectively ternarizes weights within a given packing layout based on the estimated effect of ternary conversion on the model's performance. FIONA encourages a shared ternary value within each weight group and retains full-precision weights for sensitive groups, so ternar- ized and full-precision paths coexist within a layer. It then compiles these hybrid operators exactly, applying common scaling factors once to accumulated inputs and reusing sums across outputs. Weight ternarization can also narrow the input ranges of downstream polynomials. FIONA fits lower-degree replacements under a cumulative accuracy budget, reducing multiplicative depth and bootstrapping. On VGG11, ViT, and BERT, FIONA reduces PMult operations by 53.4-79.5% and accelerates end-to-end encrypted inference by 2.38x, 1.68x, and 1.84x, re- spectively, with less than 1% accuracy loss across all three models.

1 INTRODUCTION

FIONA addresses the cost of plaintext-weight multiplication in packed FHE inference by selectively aligning ternary weights with the backend’s execution groups. It combines signed and full-precision paths to reduce encrypted inference cost while limiting accuracy loss.

1 INTRODUCTION

  • Plaintext-weight multiplication is a major FHE inference cost, and existing weight optimizations are tailored to particular encodings or architectures.Because server weights are fixed before queries arrive, offline weight changes can save operations on later inferences.Source: §1 · pp. 1–2
  • Ternary weights can replace products with addition, subtraction, or skipping, but packed execution permits this only when every weight in the group has the same ternary value.A weighted sum of 64 ciphertexts was measured to be 14.8× faster with ternary operations than with PMult.Source: §1 · p. 2
  • Scalar ternary quantization leaves only 4.06% of ResNet-8 execution groups pure and reduces CIFAR-10 accuracy from 87.53% to 82.43%.A group can avoid PMult only when its weights share one ternary value; mixed groups still require multiplication.Source: §1 · p. 2
  • FIONA uses task sensitivity to encourage a shared ternary value within each packing group, leaves sensitive groups full precision, and compiles the resulting hybrid paths.Ternarization can also narrow polynomial input ranges, allowing lower-degree replacements.Source: §1 · p. 2
  • 2.38× on VGG11, 1.68× on ViT, and 1.84× on BERT: FIONA accelerates encrypted inference with less than 1% plaintext accuracy loss across all three models.It also reduces #PMult by 53.4–79.5% and #Boot by 37.0–57.1%.Source: §1 · p. 2

2 PRELIMINARIES

These preliminaries describe encrypted inference under a stated threat model, the packed CKKS operations used to evaluate models, and the weight and polynomial representations FIONA optimizes.

2.1 ENCRYPTED DL INFERENCE WITH FHE

  • The client encrypts its input and decrypts the server’s prediction; the honest-but-curious server follows the protocol, while input shapes and padding masks are public metadata.The stated semantic-security assumption protects the input values and prediction from the server.Source: §2.1 · pp. 2–3

2.2 PACKED CKKS EXECUTION

  • CKKS packs real values into ciphertext slots for parallel operations; multiplication consumes a level, and bootstrapping refreshes ciphertexts when the available levels run out.The server replays an execution plan whose operations and schedule are fixed before each query.Source: §2.2 · p. 3

2.3 PLAINTEXT WEIGHTS UNDER A PACKING LAYOUT

  • A packing layout determines which weights share a plaintext operand, defining execution groups; ternary weights use a per-output-channel reconstruction factor to enable signed or skipped terms.Each PMult acts on the whole packed operand, regardless of the weight values inside it.Source: §2.3 · pp. 3–4

2.4 POLYNOMIAL OPERATORS

  • FHE-friendly polynomial networks replace nonlinear operators with supported arithmetic, and narrower input intervals can allow lower-degree approximations that reduce multiplication cost and depth.Lowering degree too far increases approximation error and can degrade accuracy.Source: §2.4 · p. 4

3 MOTIVATION

Plaintext-weight computation is a major FHE inference cost, but packed execution makes ternary savings depend on whole weight groups, while converting all weights can reduce accuracy. These observations motivate FIONA’s selective, packing-aware weight conversion and polynomial refitting.

3 MOTIVATION

  • Packed execution removes a PMult only when every weight in an execution group shares one ternary value, yet training toward that condition can reduce accuracy.This makes group-level uniformity and selective conversion competing design requirements.Source: §3 · p. 4

3.1 OBSERVATIONS

  • 51.0–68.0% of reported inference time goes to computation with plaintext weights across four FHE profiles, exceeding half in every profile.Figure 1 breaks down one CNN and three Transformer profiles across CPUs and GPUs.Source: §3.1 · p. 4, Figure 1 · p. 5
  • 14.16 ms for full-precision and 14.15 ms for mixed ternary groups contrast with 0.96 ms for pure ternary groups in a 64-ciphertext weighted sum.Splitting mixed groups with sign selectors takes 26.51 ms; Table 1 reports medians of five runs after one warmup.Source: §3.1 · p. 4, Table 1 · p. 5
  • Only 4.06% of 9,664 execution groups are pure after scalar ternary QAT, leaving 95.94% mixed and still requiring PMult.The ResNet-8 experiment flattened each output channel’s weights into consecutive groups of eight.Source: §3.1 · p. 5
  • 87.53% CIFAR-10 accuracy for full-precision ResNet-8 compares with 82.43% after scalar ternary QAT and 84.31% for partial conversion.Partial conversion ternarized a fixed random subset of about half the weights; 84.31% is the average over three seeds.Source: §3.1 · p. 5
  • 3.17 to 1.90 is the input-radius change after scalar ternarization, while degree-2 ternary RMSE 0.0861 is below degree-4 full-precision RMSE 0.1169.At the examined ReLU site, the narrower range permits a lower-degree polynomial at lower error.Source: §3.1 · pp. 5–6, Figure 2 · p. 6

3.2 IMPLICATIONS FOR DESIGN

  • FIONA trains toward a shared ternary value within each fixed packing group, converts groups according to task sensitivity, retains raw weights in sensitive groups, and fits polynomial degrees under an accuracy budget.It compiles the mixed paths exactly and selects polynomial degrees after weights are fixed.Source: §3.2 · pp. 6–7

3.3 POSITIONING OF OUR APPROACH

  • FIONA combines packing-group-aware selective ternarization, exact hybrid-operator compilation, and lower-degree polynomial fitting on the resulting input distributions.The approach optimizes within a given packing layout rather than requiring a different encoding.Source: §3.3 · p. 7

4 DESIGN OF FIONA

FIONA builds an offline execution plan for a fixed packing layout by routing sensitive weight groups through hybrid ternary and raw paths, compiling the resulting operators, and optimizing downstream polynomials and public-constant folds.

4 DESIGN OF FIONA

  • Given a polynomial network and fixed packing layout, FIONA uses provider training and calibration data to construct an execution plan offline.The plan proceeds through routing, exact compilation, polynomial optimization, and constant folding.Source: §4 · p. 7

4.1 PACKING-AWARE HYBRID WEIGHT ROUTING

  • Sensitivity-weighted routing encourages shared ternary values within execution groups while retaining raw weights for the most sensitive pure groups.Unprotected pure groups use ternary candidates; mixed and protected groups use raw weights.Source: §4 · p. 7, §4.1 · p. 9

4.2 EXACT COMPILATION OF ROUTED LINEAR OPERATORS

  • Exact compilation applies reconstruction factors once to signed sums, reuses common sums across outputs, and combines signed and raw results before rescaling.These algebraic rewrites preserve the layer’s function while reducing PMult, Add/Sub, and rescaling operations.Source: §4 · p. 7, §4.2 · p. 10

4.3 POST-ROUTING POLYNOMIAL OPTIMIZATION

  • FIONA fits lower-degree polynomial candidates to routed-model inputs, selecting replacements that reduce multiplicative depth while keeping accuracy within a cumulative budget.Sites are prioritized by potential depth reduction, and accepted replacements are evaluated against the same routed-model accuracy reference.Source: §4.3 · pp. 10–11

4.4 PUBLIC-CONSTANT FOLDING AND PLAN EXPORT

  • FIONA folds public constants into adjacent polynomial or linear operators, retaining the unfolded form when a numerical error check fails.It removes unused operations and exports the resulting execution plan.Source: §4 · p. 7, §4.4 · pp. 11–12

5 IMPLEMENTATION AND SETUP

The evaluation compares FIONA with raw-weight baselines on three models using the same executor and packing layout per model. It specifies training, optimization, execution, and measurement settings.

Implementation

  • The optimizer is implemented in Python with PyTorch, while encrypted inference uses a C++ executor built on Microsoft SEAL 4.1 and CKKS bootstrapping.Runs use Intel Xeon Platinum 8592+ CPUs with 48 cores and 480 GiB of memory.Source: §5 · pp. 12–13

Models and Baselines

  • The evaluation uses VGG11 on CIFAR-10, ViT-S/16 on Tiny-ImageNet, and BERT-base on SST-2, comparing FIONA with raw-weight baselines on the same architecture, executor, and packing layout.VGG11 and ViT use blocks of eight output channels and four consecutive output features, respectively.Source: §5 · p. 12

Optimization Settings

  • Optimization uses a maximum protection ratio of 0.20 and a 0.2% polynomial accuracy budget, with model-specific calibration and selection datasets.Both ViT variants use variance regularization during training.Source: §5 · pp. 12–13

Measurements

  • Latency measures end-to-end execution including layout conversions, plaintext encoding, and bootstrapping; #PMult and #Boot count operations outside bootstrapping and bootstrapping operations, respectively.Source: §5 · p. 13

6 EVALUATION

FIONA is evaluated for end-to-end encrypted inference, weight routing and exact compilation, and polynomial optimization. Across these experiments, it reduces inference cost while retaining accuracy, and its routing creates opportunities for lower-degree polynomial approximations.

6.1 RQ1: END-TO-END PERFORMANCE

  • 2.85×, 1.75×, and 3.98× plaintext-weight operator speedups accompany 37.0–57.1% fewer bootstraps and 2.04×, 2.15×, and 1.71× lower total bootstrapping time.Activation, normalization, and softmax computations also take less time; FIONA replaces some BERT attention-head Softmax exponentials with a constant, while ViT RoPE attention has no Softmax polynomial to replace.Source: §6.1 · pp. 13–14, Table 5 · p. 14
  • 2.38×, 1.68×, and 1.84× end-to-end speedups on VGG11, ViT, and BERT accompany less than 1% plaintext accuracy loss across all three models.FIONA also reduces #PMult by 53.4–79.5% and #Boot by 37.0–57.1%.Source: §6.1 · p. 13, Table 4 · p. 14

6.2 RQ2: EFFECTIVENESS OF WEIGHT OPTIMIZATION

  • 58.88% of VGG11 packed groups are pure under FIONA, reducing weight PMult counts by 58.59% versus full precision and improving accuracy by 2.23% over scalar ternary quantization.Scalar ternary quantization leaves 0.05% of packed groups pure, so mixed groups still require vector PMult.Source: §6.2 · p. 15, Table 6 · p. 15
  • On BERT/SST-2, protection raises Non-Linear accuracy from 88.07% to 91.51%; with direct transfer, the protected model scores 90.48% while the unprotected model has non-finite outputs.After refitting approximations to each model’s own inputs, accuracy is 88.30% for the unprotected model and 91.51% for the protected model, a 3.21% advantage.Source: §6.2 · pp. 15–16, Table 7 · p. 16
  • 1.60×, 1.49×, and 1.44× speedups on representative VGG11, ViT, and BERT layers result from exact compilation.Grouping reconstruction operations reduces weight PMult counts by 38.67–42.18%; sharing signed sums cuts Add/Sub counts by 4.62–8.62% without changing PMult counts. Maximum absolute error against independent FP64 computations is 5.1×10^-9.Source: §6.2 · p. 16, Table 8 · p. 16

6.3 RQ3:EFFECTIVENESS OF POLYNOMIAL OPTIMIZATION

  • 39 of 49 ViT sites and 42 of 60 BERT sites have narrower polynomial input ranges in the routed models.Mean contraction is 25.42% for ViT activations and 26.80% for BERT Softmax.Source: §6.3 · p. 17, Figure 6 · p. 17
  • 538.84× and 121.55× lower MSE occur for ViT’s routed-model quadratic activation and first-degree normalization fits than for baseline fits.For the ViT activation site, the routed-model first-degree fit has MSE 7.20 × 10^-4, below 9.13 × 10^-3 for the baseline quadratic fit.Source: §6.3 · p. 18, Figure 7 · p. 18
  • 54.07% and 49.80% reductions lower total polynomial CMult depth from 135 to 62 in ViT and from 749 to 376 in BERT.The same optimization under the same budget reduces baseline depth only to 69 in ViT and 408 in BERT; replacements occur at 34 ViT sites and 31 BERT sites.Source: Table 9 · p. 17, §6.3 · p. 18

7 RELATED WORK

FIONA uses the backend’s packing layout to guide weight optimization, then compiles hybrid weights with shared reconstruction and signed sums. It also fits lower-degree polynomials on inputs from the routed model, selecting replacements within a cumulative accuracy budget.

FHE compilation and execution

  • FIONA uses the selected packing layout to guide weight optimization and compiles the hybrid representation with common reconstruction and shared signed sums over compatible input ciphertexts.Prior systems discussed include encrypted-computation languages and IRs, automated tensor packing, convolution and level-management optimization, and packed matrix multiplication.Source: §7 · p. 18

Weight structure for FHE inference

  • Unlike approaches using block-circulant or encoding-specific weights, FIONA shapes weights toward a common ternary state within fixed packing groups and retains raw weights in mixed or protected groups.The related approaches named include SpENCNN, PrivCirNet, REDsec, and ENSI.Source: §7 · pp. 18–19

Polynomial optimization for FHE inference

  • With weights and routes fixed, FIONA fits lower-degree candidates to supplied functions on current-network inputs and selects replacements that reduce depth within a cumulative accuracy budget.The passage contrasts this with methods that jointly select polynomial degrees and bootstrap placement while adapting full-precision weights, selectively approximate non-arithmetic functions, or train with ultra-low-degree operators.Source: §7 · p. 19

8 CONCLUSION

FIONA reduces FHE inference cost through packing-aware weight optimization and lower-degree polynomials; experiments on CNN and Transformer models show speedups with little loss in plaintext accuracy.

8 CONCLUSION

  • FIONA combines packing-aware weight optimization and lower-degree polynomials to reduce FHE inference cost, with speedups and little plaintext accuracy loss in CNN and Transformer experiments.Source: §8 · p. 19

Related papers

Loading 2609.35352v1…