Source-linked AI summary
THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, Furu Wei
TL;DR
Cloud deployment of pre-trained transformers exposes sensitive plaintext user data, while homomorphic encryption does not directly support the models’ complex non-polynomial operations. THE-X approximates these operations to enable encrypted inference, achieving negligible performance reduction in the evaluated tasks while retaining theory-guaranteed privacy protection. Its approximation workflow remains constrained by accumulated LayerNorm error and sensitivity to attention-score numerical changes.
Problem
Cloud-based transformer inference risks exposing sensitive plaintext user data, while existing homomorphic-encryption operations do not directly support complex transformer functions.
Method
THE-X uses approximation components and user-device computation to replace unsupported transformer operations with fully homomorphic-compatible inference.
Results
1.49% averaged performance reduction is reported for THE-X on BERT-tiny across GLUE and CONLL2003 evaluation.
Takeaways & Limitations
THE-X enables pre-trained transformer inference on encrypted data while retaining the theory-guaranteed privacy advantage of homomorphic encryption.
Takeaways & Limitations
LayerNorm approximation error tends to accumulate as transformers stack more layers, and attention-score numerical changes can make approximation sensitive.
Abstract
from arXiv · showhide
As more and more pre-trained language models adopt on-cloud deployment, the privacy issues grow quickly, mainly for the exposure of plain-text user data (e.g., search history, medical record, bank account). Privacy-preserving inference of transformer models is on the demand of cloud service users. To protect privacy, it is an attractive choice to compute only with ciphertext in homomorphic encryption (HE). However, enabling pre-trained models inference on ciphertext data is difficult due to the complex computations in transformer blocks, which are not supported by current HE tools yet. In this work, we introduce $\textit{THE-X}$, an approximation approach for transformers, which enables privacy-preserving inference of pre-trained models developed by popular frameworks. $\textit{THE-X}$ proposes a workflow to deal with complex computation in transformer networks, including all the non-polynomial functions like GELU, softmax, and LayerNorm. Experiments reveal our proposed $\textit{THE-X}$ can enable transformer inference on encrypted data for different downstream tasks, all with negligible performance drop but enjoying the theory-guaranteed privacy-preserving advantage.
1 Introduction
THE-X targets privacy-preserving transformer inference by converting complex transformer computations into fully homomorphic operations. Its experiments report negligible performance reduction while retaining theory-guaranteed privacy protection.
- Motivation: THE-X addresses privacy risks from exposing plaintext user data during cloud-based transformer inference.Sensitive inputs can identify users and create legal, financial, and reputational risks for service providers.
- Approach: THE-X incorporates user-device computation and approximation components to support encrypted inference of pre-trained transformers.The workflow is presented as a practical conversion of fine-tuned language models into cloud services that process encrypted user data.
- Challenge: Homomorphic encryption enables computation on encrypted data, but transformer operations such as GELU are difficult to support directly.Existing HE support is limited to algebraic operations, creating a mismatch with transformer computation.
- Results: 1.49% averaged performance reduction is reported for THE-X on BERT-tiny across GLUE and CONLL2003 evaluation.The result is reported alongside privacy-preserving inference for the evaluated downstream tasks.
- Approach: THE-X designs an approximation workflow that converts transformer-based models into functions consisting of fully homomorphic operations.The work also identifies privacy-preserving inference as distinct from protecting training data.
- Results: Negligible performance reduction is confirmed in the authors’ experiments with the proposed THE-X approximation.The paper frames this result as preserving the practical privacy advantage of homomorphic encryption.
2 Background
Homomorphic encryption performs model computation on ciphertext and returns encrypted results, but existing schemes constrain transformer inference to operations that can be represented algebraically. THE-X uses a constructed encrypted-data function to address this deployment setting.
- Motivation: Pre-trained models depend on plaintext data but are difficult to apply directly to sensitive medical, search, and personally identifiable information.Prior privacy approaches include federated learning and differential privacy, but reported limitations include task coverage and performance costs.
- Homomorphic Encryption: Homomorphic encryption permits computation on encrypted data without decryption, with results decryptable to the corresponding plaintext computation.This property is intended to prevent cloud providers from accessing user plaintext during inference.
- Encrypted Inference: In the encrypted inference formulation, g is constructed to perform the role of the original model function F on encrypted data.The cloud receives ciphertext and returns an encrypted result that only the user can decrypt.
- Implementation: The Intel HE transformer backend supports CKKS through Microsoft SEAL and serves as a research tool for demonstrating HE feasibility in deep learning.The backend provides the implementation context described for the paper’s encrypted inference setting.
- HE Constraints: Partially and fully homomorphic schemes support algebraic operations whose compositions can represent polynomial functions, but not exact exponential or comparison operations.This limitation complicates direct implementation of transformer functions such as GELU and LayerNorm.
3 THE-X: Formal Description
THE-X converts transformer models into addition-and-multiplication computations compatible with homomorphic encryption by approximating non-polynomial operations and incorporating the user device into inference. Its workflow replaces GELU and softmax, distills a LayerNorm approximation, and deploys the resulting model through an HE transformer backend.
- 3 THE-X: Formal Description: THE-X combines simplified computation with user-device participation to approximate non-polynomial transformer functions under homomorphic encryption.The two core ideas are approximating non-polynomial functions and incorporating the user device into HE inference.
- 3.1.1 Gaussion Error Linear Units (GLEU): GELU is replaced by ReLU because GELU includes unsupported exponential or tanh computations, while its outputs are close to ReLU across the illustrated input ranges.The ReLU Max operation is handled through communication with the user device.
- 3.1 Approximation Workflow: The approximation workflow replaces GELU and softmax, fine-tunes the model, distills LayerNorm behavior, and removes the original LayerNorm before HE deployment.The workflow begins from a pre-trained model and produces a fully HE-supported model through staged optimization and LayerNorm distillation.
- 3.1.2 Softmax: Softmax is estimated with an addition-and-multiplication network that uses a three-layer model for reciprocal approximation and client-side ReLU computation.The estimator is trained against softmax scores using randomly generated inputs, while masked negative-infinity inputs can cause numerical problems.
- 3.1.3 LayerNorm: LayerNorm approximation retains learnable affine parameters while regressing the mean and standard-deviation-related quantities, then learns from the original LayerNorm outputs.The approximation uses γ and β and the Hadamard product; LayerNorm errors can accumulate across many transformer layers.
- 3.1.4 Other Practical Replacement: After approximation, the model contains only addition and multiplication operations and can run with the HE transformer backend, with PyTorch checkpoints first converted to TensorFlow format.Additional transformations include dropping the tanh-based pooler, replacing dense projections with fully kernel convolution, and converting matrix multiplication to element-wise operations.
- 3.2 Privacy-preserving Inference: Privacy-preserving inference splits computation between client and server: the client encrypts embeddings, the server forwards the encrypted model computation, and the client handles ReLU before returning encrypted results.All client-server communication remains ciphertext, protecting user queries from eavesdropping attackers according to the passage.
4 Experiments
Experiments evaluate THE-X with BERT-tiny across GLUE and CONLL2003 under several approximation settings and workflow choices. THE-X generally preserves performance, but layer normalization, task type, attention-mask values, weight decay, and optimization scheduling materially affect results.
- Evaluation Tasks: THE-X is evaluated on GLUE and CONLL2003 using BERT-tiny, covering sequence-level and token-level NLU tasks.The study includes classification, regression, sentence-level, and named entity recognition tasks.
- Approximation Results: 1.48% average performance reduction is observed on GLUE, with layernorm contributing 1.08% and softmax estimation only 0.09%.The average HE performance reduction is described as negligible, with sequence truncation suggested as a possible source of the slight drop.
- Approximation Results: 4.44% Pearson and 2.69% Spearman reductions occur on STS-B, whereas QNLI drops only 0.18%, indicating stronger results on classification than regression tasks.The authors assume regression tasks require higher numerical precision than classification tasks.
- Negative Infinity: Moderate attention-mask values between -2 and -5 are recommended because extreme values destabilize the softmax approximation or bias attention scores.The approximation is much more sensitive to mask-value changes than raw-model fine-tuning.
- Attention Overflow and Workflow: Attention overflow causes poor performance without regularization, while weight decay improves approximation but can hurt NLI performance; individually optimizing components is preferred.Jointly fine-tuning softmax estimation and approximated layernorm can reduce STS-B performance to 0.4%, so the two components are treated as critical stages requiring individual optimization.
5 Conclusions
THE-X enables pre-trained transformer inference under homomorphic encryption through approximation components, with slight performance cost and theory-guaranteed user privacy. The authors frame this as a first step toward addressing privacy issues in pre-trained models.
- THE-X enables pre-trained transformer models to infer under homomorphic encryption.
- Approximation components replace original transformer operations, imposing a slight performance burden while preserving theory-guaranteed user privacy.
- The work is presented as a first step toward applying homomorphic encryption to emerging privacy issues in pre-trained models.