LoREnc: Low-Rank Encryption for Securing Foundation Models and LoRA Adapters

Beomjin Ahn; Jungmin Kwon; Chanyong Jung; Jaewook Chung

Source-linked AI summary

LoREnc: Low-Rank Encryption for Securing Foundation Models and LoRA Adapters

Beomjin Ahn, Jungmin Kwon, Chanyong Jung, Jaewook Chung

arXiv:2605.13163v1May 13, 2026cs.CRcs.CVcs.LG

TL;DR

Exposed foundation-model weights and LoRA adapters create risks of unauthorized inference, extraction, and model recovery, while existing defenses may require retraining or impractical full-model decryption. LoREnc uses training-free spectral truncation and compensation, with adapter encoding, to induce structural collapse for unauthorized users and preserve authorized behavior. Experiments report strong protection, exact authorized recovery, and approximately 0.69% adapter overhead in a dense architecture.

Problem
Exposed edge-deployed foundation-model weights and structured LoRA updates enable unauthorized reuse, extraction, and model recovery, while existing protections often require retraining or impractical full-model decryption.
Method
LoREnc applies training-free spectral truncation to suppress dominant low-rank weight components, compensates them in authorized LoRA adapters, and uses secure adapter encoding.
Results
LoREnc severely degrades unauthorized inference while authorized users recover baseline outputs up to negligible floating-point errors.
Takeaways & Limitations
LoREnc provides practical FM protection that preserves authorized functionality while maintaining data-independence and efficiency for on-device deployment.
Takeaways & Limitations
Adaptive detectors specifically designed for protected adapters may still distinguish them, and formal cryptographic unrecoverability is outside the threat model.

Abstract

from arXiv · show

Foundation models and low-rank adapters enable efficient on-device generative AI but raise risks such as intellectual property leakage and model recovery attacks. Existing defenses are often impractical because they require retraining or access to the original dataset. We propose LoREnc, a training-free framework that secures both FMs and adapters via spectral truncation and compensation. LoREnc suppresses dominant low-rank components of FM weights, compensates for the missing information in authorized adapters, and further applies orthogonal reparameterization to obscure structural fingerprints of the protected adapter. Unauthorized users produce structurally collapsed outputs, while authorized users recover exact performance. Experiments demonstrate that LoREnc provides strong protection against model recovery with under 1% computational overhead.

1. INTRODUCTION

LoREnc addresses practical risks of exposing foundation-model weights by providing training-free, weight-space protection that relocates dominant spectral information into LoRA adapters. It aims to block unauthorized use while preserving authorized recovery without requiring the original dataset.

Existing defenses often require expensive retraining, reversible edge-deployed parameters, or impractical full-model decryption during inference.
LoREnc is a training-free framework that jointly protects foundation models and their LoRA adapters.
The method suppresses dominant low-rank components in foundation-model weights and compensates for them in authorized adapters.
Unauthorized access causes severe semantic degradation, while authorized users can theoretically recover the original performance exactly.
LoREnc targets model-extraction risks enabled by exposed edge weights and structured LoRA update signals.

3. PROBLEM DEFINITION AND THREAT MODEL

The threat model protects deployed foundation-model weights while preserving authorized downstream functionality through LoRA adapters. It assumes hardware-backed restoration keys and evaluates resistance to machine-learning-level extraction rather than formal cryptographic unrecoverability.

The objective is to prevent unauthorized reuse of deployed foundation-model weights while preserving authorized downstream tasks through LoRA adapters.
Encrypted model weights and adapters are accessible to adversaries, while restoration keys are assumed protected in hardware-backed environments such as TEEs.
The adversary may attempt restoration using spectral weight-extraction methods or limited fine-tuning.
LoREnc targets empirical resistance to machine-learning-level extraction, excluding physical side-channel attacks and direct key leakage.
The framework evaluates practical protection using six design requirements, including data-independence for settings where retraining or training-data collection is impractical.

4. LORENC: LOW-RANK ENCRYPTION

LoREnc truncates dominant low-rank information from foundation-model weights, compensates for it in LoRA adapters, and encrypts the resulting adapters to preserve authorized recovery while degrading unauthorized inference.

Spectral Truncation: LoREnc constructs a truncated weight by decomposing W into a deployed residual and a low-rank spectral key extracted with truncated SVD.Removing dominant singular components suppresses coherent semantic structures, leaving high-frequency residuals.
Spectral Truncation: The truncation strength is controlled by ∆r: increasing it generally improves security but increases computational overhead.
Spectral Compensation: Compensated LoRA adapters absorb the removed component so their combined weight reproduces the original downstream behavior exactly.Temporary rank expansion enables exact compensation before later adapter encryption.
LoRA Adapter Encryption: LoREnc encrypts the compensated adapters, restores their rank from r + ∆r to r, and separates encrypted weights from restoration keys.The rank reduction helps conceal whether LoREnc has been applied.
Orthogonal LoRA Reparameterization: Random orthogonal reparameterization creates equivalent adapter factorizations and removes the strict SVD-based structural fingerprint.Without it, adversaries could detect protection through structural inspection; adaptive detectors remain outside the scope of the work.
Recovery and Evaluation: Authorized reconstruction occurs on-the-fly during the forward pass, while unauthorized inference is severely degraded and authorized outputs recover baseline behavior up to negligible floating-point errors.The restored foundation-model weights are not stored on the device, and the evaluation includes CLIP, LPIPS, and autoregressive-model perplexity results.

5. EXPERIMENTS

LoREnc is evaluated across image and autoregressive models for authorized recovery, unauthorized degradation, attack resilience, and edge-device efficiency. It preserves authorized behavior, degrades unauthorized inference, resists recovery attacks, and adds less than 1% measured overhead.

Evaluation Setup: Experiments span Stable Diffusion v1.5 and autoregressive models, with additional DiT-based results supporting architecture-agnostic evaluation.The evaluation targets four questions: authorized recovery versus unauthorized degradation, fine-tuning resilience, Spectral DeTuning robustness, and edge efficiency.
Efficacy of Applying LoREnc: Authorized users recover baseline outputs up to negligible floating-point errors, while foundation-only unauthorized inference is severely degraded.Table 3 evaluates CLIP and LPIPS on SD 1.5, and Table 2 reports structurally collapsed unauthorized outputs with indistinguishable authorized outputs.
Fine-Tuning Attack: LoREnc consistently prevents meaningful foundation-model recovery after fine-tuning with 0.1k–100k samples and outperforms NNSplitter across all data regimes.Even with 100k samples, performance remains far below the original model.
Spectral DeTuning Attack: LoREnc maintains consistently high W-Error under Spectral DeTuning and large-scale adapter collection, whereas random and self-derived keys fail.Higher W-Error indicates greater deviation from original weights and therefore better protection.
Efficiency Analysis: LoREnc incurs less than 1% overhead in parameters, GFLOPs, and smartphone inference latency under real-world memory constraints.SVD is performed before deployment; inference adds only the small adapter-rank increase from r to r + ∆r.

6. CONCLUSION

LoREnc is presented as a training-free framework that secures on-device foundation models through spectral truncation and compensation. The conclusion reports structural collapse for unauthorized inference while preserving authorized integrity and satisfying requirements for practical edge deployment.

Conclusion: LoREnc employs spectral truncation and compensation to secure on-device foundation models without training.The conclusion states that the framework satisfies Effectiveness, Integrity, Resilience, Stealthiness, Data-independence, and Efficiency requirements.
Conclusion: LoREnc mathematically guarantees structural collapse for unauthorized inference while preserving integrity for authorized users.The stated design requirements include effectiveness, integrity, resilience, stealthiness, data-independence, and efficiency.
Conclusion: The framework is positioned for practical edge deployment because it maintains stealthiness, data-independence, and efficiency.These properties are listed among the six design requirements satisfied by LoREnc.

1. JUSTIFICATION OF TSVD-BASED TRUNCATION

The section formalizes why truncating the largest singular components maximizes the deviation between original and truncated weights, supporting TSVD-based perceptual encryption.

1. JUSTIFICATION OF TSVD-BASED TRUNCATION: For a matrix with non-increasing singular values, selecting the ∆r largest components maximizes the Frobenius norm of the truncated component.The maximizing index set is T = {1, . . . , ∆r}.
1. JUSTIFICATION OF TSVD-BASED TRUNCATION: The proof compares any ∆r-sized index set S with the top-index set T using σ_iⱼ ≤ σ_j for each ordered position j.This ordering follows because singular values are non-increasing and each selected index iⱼ satisfies iⱼ ≥ j.
1. JUSTIFICATION OF TSVD-BASED TRUNCATION: Taking square roots yields ∥X_S∥F ≤ ∥X_T∥F, so T is a maximizer among all subsets of size ∆r.The result directly establishes the claimed maximum deviation criterion.

2. EXPERIMENT DETAILS

The experiments evaluate LoREnc across multiple foundation models, adapters, datasets, attacks, and deployment settings, using fixed protocols and measuring both protection and efficiency.

2. EXPERIMENT DETAILS: Experiments use Stable Diffusion 1.5, GPT-2, and Llama 3 obtained from Hugging Face.Stable Diffusion uses five LoRA adapters from LoWRA Bench, while GPT-2 and Llama 3 use publicly available Hugging Face adapters.
2. EXPERIMENT DETAILS: Stable Diffusion evaluation uses the first 100 COCO validation captions, ViT-L/14 CLIP, VGG-based LPIPS, random seed 0, and default inference settings.∆CLIP is computed by subtracting baseline scores from the corresponding LoREnc-applied scores.
2. EXPERIMENT DETAILS: The study reproduces comparison procedures for NNSplitter and Spectral DeTuning, including reinforcement-learning retraining and the published W-Error protocol.The Spectral DeTuning setup uses 300 iterations with scheduled ranks from 1 to 32.
2. EXPERIMENT DETAILS: On-device measurements convert the model to LiteRT and average 50 measured runs on a Galaxy Fold 4 after 10 warm-up runs.Inference time is measured using only the UNet mid-block because of device memory constraints.
2. EXPERIMENT DETAILS: ∆r = 16 causes a non-linear jump in on-device inference latency, attributed to memory-access overheads exceeding cache thresholds or suboptimal kernel tiling.The observation motivates using ∆r = 4 as the SD 1.5 efficiency-security trade-off point.

3. ADDITIONAL QUALITATIVE RESULTS ON DIT ARCHITECTURES

Additional DiT experiments show that LoREnc’s truncation strength must match architectural spectral density: higher truncation can collapse semantics while keeping adapter overhead low.

3. ADDITIONAL QUALITATIVE RESULTS ON DIT ARCHITECTURES: LoREnc is evaluated on Sana-0.6B, a compact Diffusion Transformer selected for realistic on-device edge deployment scenarios.The evaluation protects 112 attention projection layers.
3. ADDITIONAL QUALITATIVE RESULTS ON DIT ARCHITECTURES: At ∆r = 4, Sana-0.6B retains recognizable structural concepts such as dogs and airplanes, indicating partial information leakage.The model packs dense spectral information into its weights.
3. ADDITIONAL QUALITATIVE RESULTS ON DIT ARCHITECTURES: At ∆r = 16, global structural information is completely lost, leaving only meaningless low-level features such as color.The figure visualizes the effect of varying truncation rank on generated outputs.
3. ADDITIONAL QUALITATIVE RESULTS ON DIT ARCHITECTURES: At ∆r = 16 across all 112 target layers, LoRA adapter parameter overhead remains approximately 0.69%.The result supports scaling LoREnc to architecturally dense models while preserving edge-device efficiency.

4. EFFECT OF VARYING THE ∆R ON FINE-TUNING ATTACK

The fine-tuning attack analysis varies truncation rank and dataset size to measure how truncation strength affects post-fine-tuning recoverability and overhead.

4. EFFECT OF VARYING THE ∆R ON FINE-TUNING ATTACK: CLIP scores are measured after one epoch of fine-tuning with varying dataset sizes and ∆r values.The experiment also reports the corresponding parameter and computational overhead for each truncation rank.
4. EFFECT OF VARYING THE ∆R ON FINE-TUNING ATTACK: Unless otherwise stated, the fine-tuning attack experiments use ∆r = 4.The analysis is presented as additional quantitative results for the Fine-Tuning Attack (Q2) experiment.

5. PSEUDO-CODE OF LORENC

Algorithm 1 defines LoREnc as a multi-phase pipeline that truncates dominant FM spectral structure, compensates authorized LoRA adapters, and securely encodes the resulting adapters. The procedure outputs a truncated FM, encrypted adapter factors, and restoration keys.

Evaluation: Table 3 evaluates fine-tuning attack resilience while varying delta_r on Stable Diffusion and compares the results with baseline Stable Diffusion.The table uses a specified prompt, with the baseline shown in the last row.
Inputs: LoREnc takes an original FM weight matrix, downstream LoRA adapters, and ranks delta_r and r as inputs.
Spectral Truncation: TSVD extracts the dominant low-rank structure L, which is subtracted from W to produce the deployed truncated FM W_tilde.The spectral structure is represented as L = U_FM @ diag(S_FM) @ Vh_FM.
Spectral Compensation: For each adapter, LoREnc expands the factors and injects FM spectral components using symmetrically distributed singular values S_sqrt.The compensated factors concatenate U_FM * S_sqrt with B_k and stack S_sqrt @ Vh_FM with A_k.
Secure Adapter Encoding: SVD of each compensated adapter separates restoration keys from encrypted adapter factors before secure encoding.The procedure splits the expanded factors into restoration keys K_B and K_A and encrypted factors B_enc and A_enc.
Secure Adapter Encoding: Random orthogonal reparameterization produces final encrypted factors, while the output retains the truncated FM and restoration keys.The final outputs are W_tilde, {B_final, A_final}, and {K_B, K_A}.