Source-linked AI summary
Encrypt What Matters: When Selective Homomorphic Inference Is Efficient
Ali Backour, Juan Reyes, Jaime Punyed, Ana Onoprishvili
TL;DR
Fully homomorphic inference protects private inputs but is costly when the entire input is encrypted. This paper evaluates selective homomorphic inference, encrypting only a sensitive ROI and propagating its dependencies; locality-preserving architectures achieve large speedups, whereas early global mixing provides little or no benefit.
Problem
Full-input FHE inference is expensive, motivating the question of when encrypting only a sensitive region can be efficient.
Method
The method propagates encrypted dependencies through a model, evaluates only tainted computations homomorphically, and performs independent computations in plaintext.
Results
Architectural locality governs the benefit: VGG-11 achieves 18× speedup at a 32 × 32 ROI, while architectures with early global mixing show no benefit or only 1.15×.
Takeaways & Limitations
Encrypted-dependency propagation can serve as an architectural criterion or design objective for privacy-preserving neural networks.
Takeaways & Limitations
The predictor is an architectural cost model rather than an exact runtime simulator, and non-CNN results are predictions rather than measured FHE runtimes.
Abstract
from arXiv · showhide
Fully homomorphic encryption (FHE) enables inference on private data without revealing it to the server, but evaluating an entire input under FHE is expensive. We study \emph{selective homomorphic inference}, where only a sensitive region of interest (ROI) is encrypted, and computations independent of that region are performed in plaintext. Selective evaluation produces the same output as full FHE on the same model, without retraining. Its efficiency depends on how quickly encrypted dependencies spread through the network. For small encrypted ROIs, locality-preserving architectures can achieve order-of-magnitude homomorphic-evaluation speedups, whereas architectures with early global mixing provide essentially no speedup. These results identify locality as the key architectural property governing the benefit of selective homomorphic inference.
1 Introduction
Selective homomorphic inference encrypts only sensitive input regions and evaluates dependent computations under FHE. Its efficiency depends on how quickly encrypted dependencies spread through the network.
- 1 Introduction: FHE protects private inputs during remote inference, but encrypting complete inputs makes neural-network inference substantially slower than plaintext computation.Prior approaches largely assume that the entire input is private and therefore encrypt the complete input.
- 1 Introduction: Selective inference targets spatially local sensitivity by encrypting only a region of interest while leaving unrelated input visible.Examples include sensitive faces, lesions, names, or identifiers embedded within otherwise public images or documents.
- 1 Introduction: Selective evaluation follows the dependencies of the encrypted ROI while preserving the same output as full encryption.The procedure is summarized as evaluating only ROI-dependent computations under FHE.
- 1 Introduction: Encrypted dependencies can spread beyond the original ROI, requiring activations far outside it to remain under homomorphic evaluation.Neural networks repeatedly mix their inputs, so fewer encrypted pixels do not necessarily yield proportionally less homomorphic work.
- 1 Introduction: The paper tracks encrypted-dependency propagation and finds that efficiency is governed by how long dependencies remain local, rather than by depth or model family alone.For CNNs, dependency regions grow predictably with receptive fields until saturation, and real TFHE execution validates the resulting behavior.
2 Method
The method propagates taint from an encrypted ROI through the model, evaluating tainted computations homomorphically and the rest in plaintext. Its benefit lasts only while the tainted region remains smaller than the feature map.
- 2 Method: Only activations depending on encrypted inputs require homomorphic evaluation; computations depending solely on visible inputs can remain in plaintext.This taint-based partition determines the amount of encrypted computation.
- 2 Method: The procedure propagates the ROI through the circuit, partially evaluates visible inputs, and evaluates the reduced circuit homomorphically with plaintext contributions as clear constants.This produces a reduced circuit containing only computations affected by encrypted inputs.
- 2 Method: Selective and full evaluation produce the same output under fixed quantization and lookup tables, while only the ROI values remain encrypted from the server.The ROI index set and values outside the ROI are intentionally public.
- 2 Method: For stride-1 valid convolutions, the tainted region grows with receptive field until saturation, so locality determines how long selective savings persist.The stated CNN lemma analyzes tainted-region growth and the resulting speedup.
- 2 Method: After saturation, later layers are effectively as expensive as full encryption, so a small ROI alone is insufficient.The architecture must preserve locality for enough of the network to maintain an advantage.
3 Experiments
Experiments show that selective-inference speedups track architectural locality: VGG-11 benefits strongly, while early global mixing eliminates gains. The predictor extends this comparison across additional architectures.
- 3 Experiments: The experiments use channel-narrowed AlexNet, ResNet-18, and VGG-11 variants with fixed quantization and Concrete TFHE, and all selective circuits exactly reproduce full-encryption integer outputs.Timings are medians of three homomorphic evaluations excluding compilation, key generation, encryption, and decryption.
- 3 Experiments: 18× speedup at a 32 × 32 ROI was achieved by VGG-11, versus 5.1× for ResNet-18 and 4.3× for AlexNet.The ROI covered 2.04% of the 224 × 224 input, and the ordering followed early down-sampling rather than depth.
- 3.1 Across architectures: At a 32×32 ROI, VGG-11 retained 18× speedup, Swin-T retained 2.01×, and ViT, MLP-Mixer, and a fully connected network showed no benefit.EfficientNet-B0 reached only 1.15× because global squeeze-and-excitation spread encrypted dependence across the feature map.
- 3.1 Across architectures: Architectures that preserve local dependencies retain larger gains, whereas early global mixing makes encrypted dependence global and removes the benefit.The comparison separates architectures according to how they mix spatial information.
4 Discussion
The paper proposes encrypted-dependency propagation as an architectural criterion for privacy-preserving neural networks, while selective inference balances efficiency against protecting only the encrypted ROI.
- Encrypted-dependency propagation can serve as a design or architecture-search objective for privacy-preserving neural networks.Kernel size, stride, pooling, attention windows, and global reductions determine how quickly encrypted dependencies spread.
- Smaller encrypted regions reduce homomorphic work, but only ROI values receive cryptographic protection.The practical operating point is the smallest ROI covering the sensitive content, considered jointly with dependency propagation.
5 Limitations
The evaluation has two important scope boundaries: non-CNN results are predictions, and measured runtimes come from channel-narrowed model variants rather than original full-width networks.
- The predictor is an architectural cost model, not an exact runtime simulator, so non-CNN results are predictions rather than measured FHE runtimes.The wall-clock experiments use channel-narrowed AlexNet, ResNet-18, and VGG-11 variants to make FHE evaluation tractable.
A Measurement Details
The measurements compare selective and full homomorphic evaluation under controlled hardware, software, quantization, timing, and channel-narrowed architectures. At a 32 × 32 ROI, selective evaluation substantially reduces homomorphic time while preserving outputs, but reported runtimes apply only to narrowed variants.
- Quantization and evaluation: The selective circuit separates plaintext and encrypted contributions while presenting each lookup table with the same accumulator as full evaluation.This preserves the shared quantization and lookup-table behavior used in the comparison.
- Scope: The reported absolute latencies and speedups are for channel-narrowed variants, not the original full-width networks.Channel narrowing preserves spatial dependency propagation, but limits interpretation of absolute runtime values.
- Timing protocol: Evaluation latency is the median of three executions and excludes compilation, key generation, encryption, and decryption.The excluded costs are reported separately, while compilation and key generation can be reused across requests.
- Correctness: All selective circuits reproduce the full-encryption integer output exactly under the shared fixed quantization setup.Correctness was also checked against plaintext and under real TFHE keys on held-out inputs.
- Results: 4517.7 s falls to 251.1 s for VGG-11 at a 32 × 32 ROI, yielding a 17.99× selective-evaluation speedup.These are homomorphic-evaluation times under the main comparison.
B CNN Locality and Cost Scaling
For a centered encrypted ROI in stride-1 valid-convolution stacks, the tainted region expands with receptive fields until saturation, yielding Θ(n^2/m^2) speedup scaling while locality remains.
- A stride-1 convolution expands a tainted interval by k − 1, and applying this independently across axes produces a square tainted region.This derivation assumes the expanding region has not reached a feature-map boundary.
- After i layers, the exact tainted edge reflects both receptive-field expansion and feature-map shrinkage from valid convolutions.The unsaturated expression reduces to the simpler expansion formula before the tainted region reaches the boundary.
- Saturation begins when the expanding tainted region reaches the shrinking feature-map boundary, limiting the duration of the locality benefit.The unsaturation condition determines whether the idealized speedup scaling persists through depth d.
- Θ(n^2/m^2) speedup scaling holds before saturation when receptive-field growth is small relative to both ROI and input sizes.The selective and full costs scale respectively with d m^2 and d n^2 in this regime.
C Support-Propagation Predictor and Limitations
The support-propagation predictor tracks encrypted dependencies through an architecture and estimates selective-inference speedup from tainted activation counts. It handles local and global architectural operations, with low error across measured configurations.
- Support propagation marks activations tainted when they depend on encrypted inputs, while learned weights remain structurally active and additive constants add no dependency.The resulting tainted counts drive the predicted speedup.
- Convolutions and pooling propagate support through receptive-field intersections, point-wise operations preserve it, and additions or residuals take unions of input supports.Local attention remains window-limited, whereas global token mixing can make dependence global after one participating input is tainted.
- Algorithm 2 estimates speedup by recording total activation counts with an all-ones mask and tainted counts with the encrypted-input mask.These two propagation passes provide the inputs to the predictor.
- The predictor achieved 2.7% median error and 8.3% worst-case error across nine configurations with real TFHE measurements.It also gives 1.00× predicted speedup for a whole-image ROI and no tainted computation for an empty ROI.