PaperlayerView exact revision on arXiv

Source-linked AI summary

Pacifo: An Efficient Fully Homomorphic Encryption-based Lightweight ResNet for Trustworthy Machine Learning as a Service

Yang Li, Gang He, Ruichen Zhang, Guangyuan Liu, Yinqiu Liu

arXiv:2610.33525v1cs.CRcs.AI

TL;DR

MLaaS inference can require clients to upload private data, while FHE-based CNN inference remains costly. Pacifo combines public-private layer partitioning, an efficient ReLU approximation, and flexible security configurations for ResNet20. It reports up to 21.7× speedup over existing solutions with minimal accuracy loss.

  • Problem

    FHE-based CNN inference still faces high computational overhead and runtime costs, while existing solutions typically use fixed security standards.

    Source: §I · pp. 1–2

  • Method

    Pacifo combines layer-freezing-based public-private partitioning, a low-depth ReLU approximation with knowledge distillation, and configurable security standards for ResNet20 inference.

    Source: A. Contributions · p. 2

  • Results

    Up to 21.7× speedup over existing solutions with minimal accuracy loss is reported for Pacifo.

    Source: §IX · p. 13

  • Takeaways & Limitations

    Pacifo's reported results indicate a faster FHE-based ResNet20 private-inference framework with flexible security configurations.

    Source: §IX · pp. 12–13

Abstract

from arXiv · show

Machine Learning as a Service (MLaaS) has become a popular paradigm for deploying AI applications. However, it raises significant privacy concerns that the raw client data needs to be uploaded to the server for inference services. Private inference using Fully Homomorphic Encryption (FHE) offers a solution by allowing computations directly on encrypted data, with only the final results decrypted on a client side. However, current practical implementations still suffer from large computational overhead and high latency due to the cost of FHE operations. In this paper, we propose Pacifo, a framework that adapts customized fine-tuning strategies to efficiently evaluate the ResNet20 model in the context of FHE. Specifically, Pacifo introduces a public-private layer partitioning scheme and an efficient ReLU approximation, significantly reducing the number of private layers and costly FHE operations during inference. It also offers flexible security configurations to accommodate varying client privacy requirements. Compared to state-of-the-art FHE implementations of ResNet20, Pacifo achieves a speedup of $3.7-21.7\times$ in runtime while effectively preserving $95\%-99\%$ of the original plaintext model accuracy across downstream tasks. Pacifo is open-sourced at https://github.com/BryceYangLi/Pacifo.

I. Introduction

MLaaS can expose client data because inference requires uploading inputs, while FHE-based CNN inference remains costly and offers limited security flexibility. Pacifo reduces the encrypted workload through layer partitioning and ReLU approximation, and reports faster ResNet20 inference with limited accuracy loss.

I. Introduction

  • MLaaS asks clients to transmit inputs to remote servers, raising privacy concerns around potential misuse even when the server is considered trustworthy.Private inference aims to provide model inference without exposing client data.Source: §I · p. 1
  • Existing FHE-based CNN inference incurs costly convolutions, nonlinear approximations, and bootstrapping; reported bootstrapping shares reach 72.6% and 82.3% of runtime in Lee et al. and Kim et al..Prior optimizations still leave protocols impractical for real-world applications, and they do not address flexible security standards.Source: §I · p. 2

A. Contributions

  • Pacifo partitions ResNet20 into public and private layers, using a layer-freezing fine-tuning strategy so the client computes public layers and the server evaluates encrypted intermediate features with private layers.The partition can be adjusted to trade performance against efficiency.Source: A. Contributions · p. 2
  • Pacifo approximates ReLU using only two multiplicative depths in HE and uses knowledge distillation during fine-tuning to limit performance degradation.The approach trades partial functional accuracy for lower computational overhead.Source: A. Contributions · p. 2

A. HE-only Private Inference

  • Earlier HE-only work established encrypted neural-network inference, but ResNet20 implementations remained computationally expensive, with later work reporting roughly six minutes per inference or around five minutes on a laptop.Pacifo instead emphasizes high-level circuit design to reduce the portion of the model evaluated under HE.Source: A. HE-only Private Inference · p. 3

B. Alternative Approaches to Private Inference

Alternative private-inference approaches use garbled circuits, secret sharing, or hybrid protocols, often with online interaction or multiple servers. The section also establishes notation, a semi-honest threat model, and the residual-block structure used in ResNet20.

B. Alternative Approaches to Private Inference

  • Alternative schemes include secret-sharing systems and hybrids such as Gazelle and MUSE, while Sonic uses two servers under a non-collusion assumption.Sonic's security and correctness are fragile to server collusion or compromise; the cited alternatives operate interactively, and some require multiple servers.Source: B. Alternative Approaches to Private… · p. 3
  • Pacifo's non-interactive design is distinct from these interactive alternatives and entails minimal communication overhead for client–server MLaaS deployments.The figure caption identifies a ResNet20/CIFAR-10 performance comparison, but supplies no result values.Source: B. Alternative Approaches to Private… · p. 3, Figure 2 · p. 3

A. Notations.

  • The notation defines real-valued matrices and vectors, and uses a tilde to mark a value encrypted by HE.The supplied passage also states that R and Z denote real numbers and integers, respectively.Source: A. Notations · pp. 3–4

B. Threat Model

  • The threat model is semi-honest: parties follow the protocol but may passively seek additional private information.The passage describes mutual trust as foundational and distinguishes this model from malicious protocol deviations.Source: B. Threat Model · p. 4

C. ResNet20 Model.

  • ResNet20 is described as nine residual blocks, each comprising convolutional layers, batch normalization, and ReLU activations, with shortcut connections preserving input information.The model is identified as a CNN with residual connections addressing vanishing gradients.Source: C. ResNet20 Model · p. 4

D. RNS-CKKS Scheme

Pacifo uses RNS-CKKS as a leveled FHE scheme and vector encoding to compute convolutions on encrypted feature maps. The convolution cost scales with kernel size and output channels.

D. RNS-CKKS Scheme

  • RNS-CKKS supports multiplicative depth L, and ciphertext bootstrapping refreshes a ciphertext with reduced noise when its level becomes too high.The paper uses the scheme’s homomorphic operations as a black box.Source: D. RNS-CKKS Scheme · pp. 4–5

A. Convolution

  • Vector encoding packs an input feature map into one ciphertext when cin · w · w ≤ N/2, enabling encrypted convolution through aligned rotations and kernel vectors.The approach requires k^2−1 rotations and k^2 multiplications for a filter.Source: A. Convolution · p. 5
  • The vector encoding example applies a 2×2 kernel to a 3×3 input, while the approach extends to ResNet20’s typical 3×3 convolutions on 32×32 maps.The cited extension excludes downsampling convolutions.Source: A. Convolution · p. 5
  • For a convolution with kernel size k and cout output filters, the computation uses k^2 · cout multiplications, k^2+cout−1 rotations, and k^2 · cout additions.The described complexity depends only on kernel size and output channels; downsampling uses additional masking and rotations.Source: A. Convolution · p. 5

B. Batch Normalization

Batch normalization acts on each feature-map element, and inference-time statistics allow its computation to be simplified as an affine transformation.

B. Batch Normalization

  • BatchNorm applies its transformation to each element xi of a feature map in a mini-batch.The normalization uses the batch mean and variance, a small ϵ, and trainable γ and β.Source: B. Batch Normalization · p. 5
  • At inference, the mean and variance are precomputed constants, allowing BatchNorm to be rewritten as an affine transformation Axi+B that consumes one circuit depth.The original elementwise transformation includes trainable affine parameters γ and β.Source: B. Batch Normalization · p. 5

15 End function

The circuit design combines convolution with BatchNorm to reduce depth, alongside the paper’s description of ReLU and the encrypted ConvBN operation.

15 End function

  • Combining BatchNorm with the preceding convolution saves circuit depth.Algorithm 2 describes the integrated convolution and BatchNorm computation.Source: §15 · p. 6

C. ReLU

  • ReLU is an activation function that introduces nonlinearity in neural networks.Source: C. ReLU · p. 6

15 End function

  • The ConvBN procedure takes an encrypted feature map and BatchNorm parameters and returns an encrypted output after convolution and BatchNorm.Its listed inputs also include rotation indexes, filters, and input width.Source: C. ReLU · p. 6

17 End function

Pacifo uses a scaled quadratic to approximate ReLU under FHE, relying on layer freezing and knowledge distillation to limit accuracy loss and Horner’s method to reduce evaluation cost.

17 End function

  • Because FHE supports only polynomial operations, computing max(0, x) non-interactively is nontrivial, motivating a polynomial ReLU approximation.Source: §17 · p. 6
  • The quadratic ReLU approximation has maximum error 6.06 × 10^-2, with coefficients a = 0.4950, b = 0.5000, and c = 0.0656.Inputs are scaled into [−1, 1], and the coefficients are selected using the Remez exchange algorithm.Source: §17 · p. 7
  • Layer freezing lets most ReLU activations be computed accurately in plaintext, while knowledge distillation mitigates accuracy loss from approximating the remaining activations.Source: §17 · p. 7
  • Horner’s method evaluates the quadratic as (ax + b)x + c, saving one multiplication versus direct evaluation as ax^2+bx+c.Source: §17 · p. 7

5 End function

Pacifo reduces FHE costs through carefully budgeted bootstrapping and a residual-block circuit design. Its layer-partition approach further limits which model computations run under HE.

D. Design of bootstrapping

  • Two bootstrapping operations per residual block, each budgeted at 15 levels, balance bootstrapping efficiency with fewer invocations.The design allocates {3, 3} levels to CoeffToSlot and SlotToCoeff and nine more to ModRaise, Chebyshev approximations, and double-angle iterations.Source: D. Design of bootstrapping · p. 7

E. Overall Circuit

  • Algorithm 4 generalizes the FHE operations for one ResNet20 residual block, which can be stacked to evaluate the full model.The design is aligned with Fig. 5's bootstrapping placement.Source: E. Overall Circuit · p. 7, Figure 5 · p. 7

V. Fine-Tuning based Layer Partition

  • Pacifo's architecture-level approach minimizes layers evaluated under HE, while transfer learning is used to maintain accuracy after partitioning.Source: §V · p. 7

10 End function

Layer partitioning assigns early model layers to client-side plaintext inference and later layers to server-side encrypted inference. This reduces HE computation while limiting which parameters can be updated.

A. Layer Partition for Efficiency

  • Restricting fine-tuning to final private layers leaves unchanged layers public, based on the premise that updating only a subset of pretrained parameters can perform well.The cited prior approaches apply this idea in MPC or add layers for a simple binary classification task.Source: A. Layer Partition for Efficiency · p. 8
  • The client computes public layers, encrypts the intermediate representation, and sends it to the server for private-layer inference; the client then decrypts the prediction.With r public blocks and T−r private blocks of similar HE cost, the stated overhead reduction is a factor of T/(T−r), saving r/T × 100% of homomorphic computation.Source: A. Layer Partition for Efficiency · p. 8
  • Client-side plaintext inference adds computation, but the authors describe its overhead as minimal compared with encryption and decryption.Source: A. Layer Partition for Efficiency · p. 8

B. Transfer learning for Accuracy

Transfer learning helps preserve performance when layer partitioning restricts fine-tuning to a small private portion of the model. Pacifo also offers multiple HE security levels, and its security analysis assumes a static semi-honest adversary.

B. Transfer learning for Accuracy

  • A small private portion limits trainable parameters and can degrade adaptation to new tasks; transfer learning initializes from a related source dataset and freezes public layers while updating private layers.The paper says this pairing complements efficiency from layer partitioning with performance preservation under limited trainable parameters.Source: B. Transfer learning for Accuracy · pp. 8–9

VI. Customized Security Levels

  • Pacifo provides flexible HE parameter schemes for multiple security levels, allowing clients to trade off performance and security; its schemes support at least 128-bit security, with 192-bit and 256-bit options.Source: §VI · p. 9

VII. Security Analysis

  • The security analysis uses a simulation-based paradigm under a static semi-honest adversary model, in which either the client or server may be corrupted.Source: §VII · p. 9

B. Security Proof of Pacifo

Pacifo is stated to securely realize secure inference against a static semi-honest adversary, assuming semantic security of its RLWE-based FHE scheme. The proof invokes correctness of homomorphic operations and simulation-based arguments for client and server views.

B. Security Proof of Pacifo

  • Under semantic security of the RLWE-based FHE scheme, Pacifo securely realizes secure inference against a static semi-honest adversary.Source: B. Security Proof of Pacifo · p. 10
  • Correctness follows from the underlying homomorphic operations, which produce the correct inference result in encrypted form.Source: B. Security Proof of Pacifo · p. 10
  • For a corrupted client, the proof simulates encrypted model parameters with encryptions of zero, relying on semantic security to make the views indistinguishable.Source: B. Security Proof of Pacifo · p. 10
  • For a corrupted server, the proof simulates encrypted client inputs with encryptions of random plaintexts and forwards the encrypted inference output.Source: B. Security Proof of Pacifo · p. 10

VIII. Experimental Results

Pacifo’s evaluations combine layer freezing and knowledge distillation with efficient ReLU approximation to reduce FHE inference costs while preserving accuracy. The macro-benchmarks report faster ResNet20 inference, low communication overhead, and ciphertext accuracy and precision comparable to plaintext.

A. Implementation

  • The implementation uses OpenFHE CKKS and HEXL on an Intel Xeon Gold 5218R CPU, with 128-bit security and different circuit parameters for full-layer and block circuits.The full ResNet20 layer circuit uses N = 2^16 and L = 24; the block circuit uses N = 2^15 and L = 17.Source: A. Implementation · p. 10

B. Models and Datsets

  • Pacifo{L} and Pacifo{B} are formed by freezing the first six and eight residual blocks of a ResNet-20 pretrained on CIFAR-100, then using a teacher model for downstream-dataset knowledge distillation.Pacifo{L} retains more trainable parameters and learning capacity but incurs greater inference cost than Pacifo{B}.Source: B. Models and Datsets · p. 10

C. Micro-benchmark Evaluation of ReLU

D. Macro-benchmark Evaluations

  • 21.7× and 25.2× runtime speedups are reported for Pacifo{B} over Kim et al. and Rovida et al., with 2% accuracy degradation versus Kim et al..Pacifo{L} reaches 92.58% accuracy and is 2.7× and 3.1× faster than Kim et al. and Rovida et al., respectively.Source: D. Macro-benchmark Evaluations · p. 11, Table VI · p. 12
  • For a basic residual block of Pacifo{L}, Conv+BatchNorm and bootstrapping are the most time-consuming components, taking 9.39 s (50.75%) and 7.97 s (43.07%) of total runtime, respectively.In a basic block in layer 3, the optimized ReLU implementation contributes less than 0.05% to overall execution time.Source: D. Macro-benchmark Evaluations · p. 11, Table VII · p. 12
  • Pacifo{L} communication is roughly 11.5MB and Pacifo{B} communication is 10.5MB, with one ciphertext sent by the client and one returned by the server.With N = 2^15, the output size peaks at 16384 in layer 1, allowing an arbitrary block output to fit in one ciphertext.Source: D. Macro-benchmark Evaluations · p. 11, Figure 7 · p. 11
  • Ciphertext inference accuracy is comparable to plaintext across MNIST, CIFAR-10, and CIFAR-100, and relative precision remains above 0.98 throughout inference.Relative precision gradually declines as FHE noise accumulates.Source: D. Macro-benchmark Evaluations · pp. 11–12, Table VIII · p. 12

E. Scalability on Tuning HE Parameters.

Pacifo{L} is evaluated under higher FHE security settings to examine the runtime trade-off. Its runtime rises from the 128-bit configuration as security increases, while the conclusion reports speedups of up to 21.7× over existing solutions with minimal accuracy loss.

E. Scalability on Tuning HE Parameters.

IX. Conclusion

  • The conclusion reports up to 21.7× speedup over existing solutions with minimal accuracy loss and scalability to higher security levels.Pacifo combines layer partitioning, lightweight ReLU approximation, and flexible security configurations.Source: §IX · pp. 12–13
Loading 2610.33525v1…