PaperlayerView exact revision on arXiv

Source-linked AI summary

An Open-Source End-to-End FHE Implementation for Privacy-Preserving Llama 3 8B Inference

Yuhang Fan, Yusi Chen, Kanyu Ye, Zhuoran Ji

arXiv:2609.12378v2cs.CR

TL;DR

Cloud LLM inference can expose sensitive prompts to providers, and FHE packing affects the costs of computation and data movement. Odin co-designs packing and execution for Llama-3, combining cross-layer and operator layouts with nonlinear approximation and error allocation. On one H100, it evaluates all 32 layers in 366.4 s, compared with THOR’s 1651.9 s under the stated matching setup.

  • Problem

    Cloud LLM inference gives providers access to prompts that may contain sensitive information, while FHE inference efficiency depends on packing choices that affect ciphertext and encoding costs.

    Source: §1 · p. 1, §3.4 · p. 7

  • Method

    Odin co-designs a feature-major cross-layer layout and transient operator layouts with calibrated minimax approximations and model-level joint error allocation.

    Source: §1.2 · p. 2, §5.1 · p. 7, §5.2.3 · p. 9

  • Results

    366.4 s versus THOR’s 1651.9 s for 32-layer Llama-3-8B inference with a 128-token input on one NVIDIA H100 80 GB GPU, a 4.51× speedup.

    Source: §7 · p. 13

  • Takeaways & Limitations

    Odin demonstrates an open-source end-to-end GPU CKKS inference path for Llama-3-8B under the reported input and hardware setting.

    Source: §1.2 · p. 2, §7 · p. 13

    The error-injection search selects candidate budgets but does not guarantee that the final Remez polynomials produce the same PPL.

    Source: §5.2.2 · p. 8

Abstract

from arXiv · show

Cloud LLM services typically require users to send prompts to a model provider, creating a privacy risk. Fully homomorphic encryption (FHE) lets a server perform inference without decrypting the input, but representing data as ciphertexts adds storage and computational overhead. In CKKS-based LLM inference, the packing scheme maps logical tensors to ciphertexts and slots. It therefore determines the ciphertext count and the homomorphic cost of linear layers, and it constrains how data pass between linear layers, attention, and nonlinear computation. As models and sequences grow, inefficient layouts accumulate encoding, compute, and layout-conversion overhead. We present Odin, an FHE inference system that co-designs ciphertext packing and model execution for Llama. Starting from a THOR-style baseline whose bottleneck is weight encoding, Odin uses a feature-major cross-layer layout to unify residual connections and layer interfaces, and builds transient intra-operator layouts for linear projections and attention. This reduces redundant plaintext encoding of weights in wide projections. Within attention, QK^T produces scores that Softmax can consume directly, and PV consumes the resulting probabilities, avoiding intermediate repacking. For nonlinear ops, we use minimax polynomial approximation with input-range control and joint error allocation guided by model quality, reducing polynomial degree and multiplicative depth. To our knowledge, Odin is the first open-source end-to-end GPU CKKS implementation of Llama-3. With Llama-3-8B weights and a 128-token input, Odin evaluates all 32 Transformer layers on a single NVIDIA H100 80 GB GPU. Server-side end-to-end FHE evaluation takes 366.4 s and 58.9 GiB peak device memory. Under the same model, input, CKKS parameters, and hardware, THOR takes 1651.9 s, a 4.51x speedup.

1 Introduction

Cloud-hosted LLM inference exposes user prompts to providers, while FHE offers a way to compute over encrypted inputs. Odin co-designs packing and execution for Llama, addressing weight-encoding costs and connecting operator layouts into an end-to-end inference path.

1.1 Related Work

  • MPC and hybrid systems protect inputs but generally require multi-round client or server interaction and transmit large volumes of intermediate data.Their security also depends on assumptions such as non-collusion among participating parties.Source: §1 · p. 1, §1.1 · p. 2

1. Introduction

  • Ciphertext packing affects ciphertext counts, linear-operation costs, weight encoding, and layout-conversion costs as model dimensions and depth grow.FHE-only systems have advanced to complete Transformers, but packing formats must remain compatible across the computation pipeline.Source: §1 · pp. 1–2, §1.1 · p. 2

1.2 Our Approach and Contributions

  • Odin combines a feature-major cross-layer interface with transient layouts for projections and attention, reusing encoded weight diagonals in wide projections.This design follows a THOR-style Llama-3 baseline whose linear-computation bottleneck is online weight encoding.Source: §1.2 · p. 2
  • QK⊤ scores pass directly to Softmax, whose probabilities pass directly to PV, avoiding intermediate repacking between Softmax and PV.The system also uses input-range control, polynomial approximation, and model-level error injection to search candidate nonlinear error budgets.Source: §1.2 · p. 2
  • Odin uses the actual Llama-3-8B weights to evaluate all 32 layers on one H100 in 366.4 s, using 58.9 GiB peak memory.The implementation includes model conversion, GPU ciphertext execution, and experimental configurations.Source: §1.2 · p. 2

2 Background

The background describes CKKS ciphertext computation and the Llama-3-8B operations that must be represented within it. In the stated execution model, weights remain plaintext while activations are encrypted.

2.1 The CKKS Computation Model

  • CKKS packs vectors into ciphertext slots for SIMD operations; multiplication and rescaling lower ciphertext level, while bootstrapping restores the level budget.Decoding after decryption recovers an approximation to the encoded vector.Source: §2.1 · p. 3
  • The execution uses Encode, Add, plaintext–ciphertext and ciphertext–ciphertext multiplication, slot rotations, key switching, rescaling, and bootstrapping.Bootstrapping restores level budget but adds latency and approximation error.Source: §2.1 · p. 3

2.2 The Llama-3-8B Architecture

  • Llama-3 uses RMSNorm pre-normalization, RoPE, grouped-query attention, and a SwiGLU FFN; encrypted activations make projections plaintext–ciphertext operations and QK⊤ and PV ciphertext–ciphertext operations.RMSNorm, Softmax, and SiLU require polynomial or iterative CKKS circuits.Source: §2.2 · p. 3

3 FHE Llama-3: Problem Definition and Baselines

The system assumes a client encrypts prompts for server-side inference under an honest-but-curious threat model. Its nonlinear circuits use calibrated-interval polynomial approximations and jointly considered input ranges and error budgets.

3.1 Private Inference and Threat Model

  • The client encrypts the prompt and sends it with evaluation keys; the server evaluates encrypted activations and returns a ciphertext, under an honest-but-curious-server assumption.The scope excludes malicious-server deviations, side channels, denial of service, access-pattern leakage, and protection of model weights.Source: §3.1 · p. 3

3.2.1 Polynomial Approximation

  • The method approximates scalar nonlinear functions directly over calibrated intervals with minimax polynomials, and Softmax composes them in a doubling iteration.It avoids comparison-based domain partitioning; interval and error tolerance jointly affect the required degree.Source: §3.2.1 · pp. 3–4

3.2.2 Input-Range and Outlier Mitigation

  • Calibration initializes nonlinear input intervals, but corresponding operator positions can deviate and calibration does not guarantee all inference inputs remain within range.Earlier outlier-mitigation approaches address attention sinks along the token dimension and dimensional outliers along the feature dimension [12]; the latter uses rotations.Source: §3.2.1 · p. 4, §3.2.2 · p. 4

3.2.3 Approximation-Error Budget

  • Error allocation first groups tolerances by nonlinear operator type, then screens candidate budgets with bounded error injection and a joint search across four classes.Injected perturbations proxy sensitivity but do not ensure the final Remez polynomials produce the same PPL, so model quality is evaluated separately.Source: §3.2.3 · p. 5
  • Relaxing SiLU tolerance from 2^-12 to 3.82 × 10^-3 reduces its summed polynomial degree across 32 layers from 2,048 to 998, allowing fewer limbs in the subsequent FFN linear layer.Under the current schedule, reducing RMSNorm or Softmax degree does not necessarily reduce bootstrapping operations.Source: §3.2.3 · p. 5

3.3 Linear Computation and Packing Baselines

CKKS packing and layout compatibility shape both the cost of linear computation and the conversions required between operators. In the THOR-style Llama-3 baseline, weight packing and encoding dominate linear runtime, motivating reuse-aware layouts.

3.3.1 Matrix Representations and Linear Computation in CKKS

  • CKKS slot encoding supports slot-wise arithmetic, but a complete linear algorithm must also specify token–feature mapping, weight organization, rotations, reductions, and output layout.Packing lifetime across an operator chain matters because independently chosen layouts can require extra rotations, masks, or repacking.Source: §3.3.1 · p. 5

3.3.2 THOR Packing

  • THOR organizes plaintext–ciphertext and ciphertext–ciphertext matrix multiplication around cyclic diagonals, using rotations, slot-wise products, and accumulation.Its multi-diagonal batched encoding interleaves diagonals and packs adjacent batched diagonals into a ciphertext; its workflow also uses transposition, replication, and masking.Source: §3.3.2 · pp. 5–6

3.4 The Encoding Bottleneck at Llama-3 Scale

  • THOR's weight-diagonal interface does not specify how plaintexts are generated, cached, or reused across projection blocks and batches.In the Llama-3 adaptation, reuse requires matching physical-layout attributes, yielding 425,984 plaintext weight objects per layer and limiting amortization across input ciphertexts.Source: §3.3.2 · p. 6, §3.4 · p. 7
  • 38.02 s of plaintext–ciphertext and 6.78 s of ciphertext–ciphertext linear computation total 44.80 s, or 75.6% of one layer's runtime.The profile covers a single Llama-3-8B layer using THOR packing.Source: §3.4 · p. 6, Table 1 · p. 6
  • 44.10 s of linear computation includes 28.02 s for weight packing and encoding (63.5%); encoding alone takes 17.03 s (38.6%).The independent profile identifies encoding as the largest individual linear-computation operation.Source: §3.4 · p. 6, Table 2 · p. 6

5 System Implementation of FHE Llama

The implementation transforms the public Llama-3 model, prepares nonlinear approximations and numerical settings, and assembles a layer-wise FHE execution graph. Its artifacts are fixed before correctness, quality, and performance evaluation.

5 System Implementation of FHE Llama

  • The implementation transforms the public Llama-3 model and calibrates nonlinear input ranges before generating approximations and numerical configurations.The process then connects projections to homomorphic linear kernels and assembles a layer-wise execution graph.Source: §5 · p. 7
  • The resulting artifacts include transformed weights, measured input intervals, polynomial coefficients, and bootstrapping positions.These artifacts are fixed before evaluation of correctness, model quality, and performance.Source: §5 · p. 7
  • The execution graph is evaluated for correctness, model quality, and performance after its artifacts have been fixed.Source: §5 · p. 7

5.1 Model Preprocessing and Input-Range Calibration

The preprocessing applies QuaRot to public Llama-3 weights and calibrates nonlinear approximation intervals from observed values. A 1.2 expansion factor adds a safety margin, with a corresponding increase in polynomial degree for a fixed error target.

5.1 Model Preprocessing and Input-Range Calibration

  • QuaRot is applied to public Llama-3 weights, with Hadamard transforms before the O and down projections fused into adjacent weights.The fused transforms add no homomorphic computation to FHE execution.Source: §5.1 · p. 7
  • The transformed model is used consistently for interval measurement, polynomial generation, and FHE linear computation.This keeps calibration and actual execution on the same model.Source: §5.1 · p. 7
  • Approximation intervals use observed extrema expanded by 1.2; the safety margin reduces out-of-bound risk but increases polynomial degree for a given error target.SiLU uses a symmetric interval, while the RMSNorm inverse-square-root input uses a positive interval.Source: §5.1 · p. 7

5.2 Nonlinear-Function Approximation and Error Calibration

Odin calibrates nonlinear approximations for Llama by controlling input ranges and allocating error budgets according to model-level quality. It combines a decomposed homomorphic Softmax, joint error search, and polynomial generation, while treating CKKS numerical error separately.

5.2 Nonlinear-Function Approximation and Error Calibration

  • The section retains layer-specific approximation intervals but shares one output-perturbation budget across layers for each operator type, reducing the search to four coupled variables.Those variables correspond to RMSNorm1, Softmax, RMSNorm2, and SiLU.Source: §5.2 · p. 7

5.2.1 Decomposed Approximation of Softmax

  • Homomorphic Softmax uses an exponential polynomial followed by repeated scale-doubling rounds, with calibrated bounds and ranges guiding the approximation.A wide track handles score coordinates while a compact auxiliary track handles reductions and inverse square roots; the latter confines bootstrapping to compact ciphertexts.Source: §5.2.1 · pp. 7–8

5.2.2 Approximation-Error Proxy and Uniform Tolerance

  • Bounded uniform error injection estimates model sensitivity and selects candidate budgets, but does not reproduce Remez residuals or guarantee the same perplexity as the final polynomials.Final polynomial quality is evaluated separately after polynomial generation.Source: §5.2.2 · p. 8
  • At an injection-test perplexity increase limit of 0.001, the candidate common tolerance reaches 2.258×10−3, or 9.25 times 2−12.The authors interpret this as evidence that the 12-bit reference is conservative for bounded perturbations under the current protocol.Source: §5.2.2 · p. 9, Table 3 · p. 9

5.2.3 Cost-Aware Joint Error Allocation

  • The joint search relaxes SiLU, RMSNorm2, RMSNorm1, and Softmax successively to conditional upper bounds, with the search order prioritizing computational cost.SiLU is searched first because its polynomial depth can affect both its evaluation and the subsequent wide linear layer.Source: §5.2.3 · p. 9

5.2.4 Polynomial-Coefficient Generation

  • For each operator, layer, and Softmax substep, the coefficient generator searches polynomial degree over the calibrated interval and checks the maximum error at extrema.The generated coefficients must still be verified using complete operator outputs and model perplexity because budgets come from an error-perturbation proxy.Source: §5.2.4 · p. 10
  • Across 32 layers, joint budgets reduce summed SiLU polynomial degrees from 2048 to 736 and summed multiplicative depths from 199 to 155 versus a uniform 12-bit target.These reductions are under the current approximation intervals and evaluation strategy.Source: §5.2.4 · p. 10

5.2.5 Treatment of CKKS Noise

  • The budget search does not treat CKKS numerical error as an independent variable; its incremental effect is measured separately by comparing Poly-only and FHE execution.The single-operation CKKS error measurement is not used as a substitute for evaluating cumulative error over the complete model.Source: §5.2.5 · p. 10

6 Evaluation

Odin is evaluated against THOR using Llama-3-8B and matched model, input, CKKS, and hardware settings. The evaluation reports lower linear and full-model runtimes, reduced nonlinear computation under joint error allocation, and model-quality measurements for polynomial execution.

6.1 Experimental Setup

  • Experiments use Meta-Llama-3-8B with 128-token inputs on one NVIDIA H100 80 GB GPU, with THOR and Odin matched for model weights, CKKS parameters, and bootstrapping configuration.The CKKS ring dimension is N = 216, and the modulus chain has depth 37.Source: §6.1 · p. 10

6.2 Linear-Computation Performance

  • 5.32 s with Odin versus 41.37 s with THOR is a 7.8× speedup for Layer 0 plaintext–ciphertext computation.Table 5 covers QKV, WO, gate/up, and down projections; the comparison includes packing/encoding, homomorphic evaluation, and rescaling.Source: §6.2 · pp. 10–11, Table 5 · p. 10

6.3 Ablation of Nonlinear Precision Allocation

  • Under the tested settings, joint allocation reduces total nonlinear polynomial degree from 23,157 to 22,694 and multiplicative depth from 1,314 to 1,290.Across the four nonlinear stages participating in error allocation, runtime falls from 0.49 s at uniform 12-bit precision to 0.23 s with joint allocation.Source: §6.3 · p. 11, Table 7 · p. 11, Table 8 · p. 11

6.4 Single-Layer and Full-Model Performance

  • Odin reduces intermediate-layer linear-stage time from 38.38 s to 5.65 s and complete-layer time from 60.87 s to 11.12 s.The full-path comparison includes linear projections, layout conversions, and implementation-dependent auxiliary work.Source: §6.4 · p. 12, Table 9 · p. 12
  • 366.4 s for Odin versus 1651.9 s for THOR gives a 4.51× speedup for end-to-end online computation of all 32 layers.The comparison uses the same model, 128-token input, CKKS parameters, and hardware.Source: §6.4 · p. 12

6.5 Correctness and Model Quality

  • On eight WikiText-2 samples totaling 1,016 tokens, uniform 12-bit and joint-allocation Poly-only paths both have PPL 12.79, versus 12.67 for Exact.Their ∆PPL values relative to Exact are 0.122 and 0.119, respectively; the supplied passage does not provide final FHE PPL values.Source: §6.5 · p. 12, Table 10 · p. 12

6.6 Comparison with Existing FHE LLM Systems

  • Odin’s 128-token Llama-3-8B prefill runs on one H100 at 366.4 s, while cross-paper latencies are not directly comparable because models, encrypted-input ranges, and hardware differ.The comparison table includes THOR and MOAI on BERT-base and a Llama-2-7B system encrypting the final 128 tokens of a 4,096-token input.Source: §6.6 · p. 12, Table 11 · p. 12

7 Conclusion

Odin combines cross-layer and short-lived operator layouts with weight-diagonal reuse and polynomial nonlinear computation to build a complete CKKS inference graph for Llama-3. On Llama-3-8B with a 128-token input, it evaluates all 32 layers in 366.4 s, a 4.51× speedup over THOR.

7 Conclusion

  • Odin combines a feature-major cross-layer interface with specialized short-lived layouts to reuse encoded weight diagonals and avoid intermediate attention repacking.It also uses input-interval calibration, minimax polynomial approximation, and model-level joint error allocation for RMSNorm, Softmax, and SiLU.Source: §7 · p. 13
  • 366.4 s versus 1651.9 s for THOR yields a 4.51× speedup for the complete 32-layer ciphertext computation.The result is for Llama-3-8B with a 128-token input on one NVIDIA H100 80 GB GPU; peak device memory is 58.9 GiB.Source: §7 · p. 13

Related papers

Loading 2609.12378v2…