PaperlayerView exact revision on arXiv

Source-linked AI summary

FISGuard: Defending Against Membership Inference via Fixed Input Subspaces

Haocheng Jiang, Hua Shen

arXiv:2608.27836v1cs.CRcs.AIcs.DC

TL;DR

Federated parameter-efficient fine-tuning can leak membership through gradient geometry, while existing defenses may not directly address ProjRes. FISGuard fixes a low-dimensional input subspace using independent public data and restricts private updates to it. Across the evaluated settings, it brings ProjRes AUC near random guessing while keeping task performance close to the undefended model with limited overhead.

  • Problem

    ProjRes can infer training-set membership from projection residuals in server-visible gradients, exposing privacy despite clients sharing updates rather than raw data.

  • Method

    FISGuard constructs a fixed low-dimensional input subspace from independent public data and restricts trainable linear updates to that subspace.

  • Results

    FISGuard reduces ProjRes AUC to near the random-guessing level in most settings while maintaining downstream performance close to the undefended model and adding limited computational overhead.

  • Takeaways & Limitations

    FISGuard offers an effective privacy–utility trade-off against projection-residual membership inference across Adapter and LoRA fine-tuning.

Abstract

from arXiv · show

As large language models are increasingly adopted in federated learning, protecting user privacy while performing parameter-efficient fine-tuning on distributed private data has become an important challenge. Although clients only share gradients instead of directly uploading raw data, the shared gradients may still leak membership information about training samples. ProjRes (S&P, 2026) further increases this risk: with less information and without accessing model outputs, an attacker can effectively distinguish members from non-members solely based on the projection residual between a candidate representation and the subspace induced by server-observable gradients. Existing defenses against membership inference mostly rely on gradient perturbation or regularization, which can not only degrade model utility but also fail to effectively defend against the membership inference attack introduced by ProjRes, which exploits the geometric structure of gradients. To address this issue, we propose FISGuard, a lightweight defense. Its key idea is to construct and fix a low-dimensional representation subspace using independent public data, thereby restricting the space through which private representations are exposed via gradients while preserving the primary information required for downstream tasks. This substantially reduces the projection-residual discrepancy between members and non-members. We evaluate FISGuard against five representative defense methods across three NLP datasets, two LLMs, and two fine-tuning strategies, Adapter and LoRA. The results show that FISGuard reduces the ProjRes attack AUC to near the random-guessing level of 0.5 in most settings, while maintaining downstream task performance close to that of the undefended model and introducing only limited computational overhead, thereby achieving a favorable privacy--utility trade-off.

I. INTRODUCTION

The paper identifies a structural membership-inference risk in federated parameter-efficient fine-tuning and proposes FISGuard, which fixes a low-dimensional input subspace before private training. This removes direct exposure of original representations and constrains observable gradients so the attack’s projection residuals disappear.

  • Motivation: ProjRes infers membership from projection residuals between candidate representations and subspaces extracted from server-visible gradients.The attack exploits representations contributing to the gradient row space of trainable linear layers.
  • FISGuard: The framework removes original input-side gradients and constrains remaining gradients to a fixed low-dimensional space that can be fully covered.When both member and non-member representations are fully represented, ProjRes lacks usable projection residuals for membership ranking.
  • FISGuard: FISGuard decomposes each linear update as ∆W = UQ, freezing the input projection Q while training only the output mapping U.The projection is determined before private training and maps representations into a fixed low-dimensional coordinate space.
  • Scope and implementation: FISGuard generalizes across trainable fully connected mappings, with Fixed-Down Adapter and LoRA serving as its primary implementations.The input-side factor is fixed while the output-side factor remains trainable in both architectures.
  • Design choices: Independent public data construct the fixed projection, while private data train only the mapping from projected coordinates to task outputs.Function-preserving initialization leaves the pretrained model’s initial forward behavior unchanged.
  • Advantages: FISGuard targets gradient geometry without random noise, loss changes, aggregation changes, or inference-interface modifications.In the Adapter implementation, public data construct the fixed Down projection and only the Up projection and task head are trained.
  • Motivation: ProjRes requires direct exposure of private representations and incomplete coverage of candidate representations by the gradient space.These conditions make member representations more likely to lie inside the observed subspace than non-member representations.

II. PRELIMINARY

Federated LLM fine-tuning distributes a pretrained model to clients, who optimize local data and return updates for server aggregation. PEFT reduces this process to a small trainable parameter set, including Adapter and LoRA components.

  • Federated training: Federated LLM fine-tuning repeatedly distributes a global model, performs local training on private datasets, uploads updates, and aggregates them on the server.The resulting global model is redistributed for the next communication round.
  • Federated training: The federated objective optimizes a global model across client objectives without centralizing local datasets.Each client contributes a local training objective based on its private data.
  • Parameter-efficient fine-tuning: PEFT decomposes model parameters into frozen pretrained parameters θ and a small trainable set ϕ for downstream adaptation.Only ϕ participates in local optimization and federated aggregation.
  • Adapter and LoRA: Adapters insert lightweight bottleneck modules while keeping the pretrained backbone frozen.The Adapter uses trainable Down and Up projections around a nonlinear transformation.
  • Adapter and LoRA: LoRA represents the update as W = W0 + ∆W = W0 + BA, where A and B are low-rank trainable matrices.Only A and B are optimized and communicated during federated fine-tuning.

III. RELATED WORK

Existing membership-inference defenses mitigate overfitting, perturb updates, limit exposed information, or alter aggregation assumptions. ProjRes exposes a distinct gradient-subspace risk that these approaches do not directly eliminate.

  • Defense categories: Existing defenses address membership leakage through regularization, differential privacy, output protection, perturbation, sparsification, or secure aggregation.These methods target overfitting, exposed predictions, model parameters, updates, or client observations.
  • ProjRes: ProjRes infers membership from a single round of gradients by relating hidden representations to subspaces formed by client-uploaded gradients.The attack applies to Adapter and LoRA fine-tuning and relies on gradient geometry rather than final outputs.

IV. FISGUARD

FISGuard reparameterizes vulnerable client-trained linear layers as fixed public input projections followed by trainable output mappings. This confines private representations to a fixed low-dimensional space while preserving the standard federated workflow.

  • Target-layer identification: FISGuard identifies client-trained linear layers that receive private representations and expose updates to the server.The criterion applies across adapters, LoRA, attention projections, feed-forward layers, and task heads.
  • Fixed input subspaces: The server constructs each fixed input projection from dominant representation directions extracted by SVD on independent public data.The public data determine the allowed input directions before private training begins.
  • Security rationale: Because both member and non-member representations occupy the same fixed projected space, the remaining gradient span can cover them more completely and weaken ProjRes residuals.FISGuard changes parameterization rather than the client loss, server aggregation rule, or basic federated optimization workflow.
  • Architecture instantiations: Adapters and LoRA reduce to the common task-specific update form ∆Wℓ = UℓQℓ.Fixed-Down Adapter and fixed-input LoRA are concrete implementations of the broader construction.
  • Private training: Private training optimizes only the output-side mappings, so private representations no longer directly train or upload input-side mappings.For target layers, clients compute zℓ = Qℓxℓ and update Uℓ while Qℓ remains fixed.

V. EXPERIMENTS

The experiments were conducted on a workstation with a 13th-generation Intel Core i5-13600KF CPU, an NVIDIA RTX 3090 Ti GPU, and 64 GB of system memory.

  • Hardware and software: Experiments used Windows 11, a 13th-generation Intel Core i5-13600KF CPU, and an NVIDIA GeForce RTX 3090 Ti GPU with 24 GB VRAM.The workstation had 64 GB of system memory.

A. Experimental Setup

The evaluation covers three NLP classification datasets, two Transformer language models, and five representative membership-inference defenses under a common federated training configuration.

  • Datasets: The datasets are CoLA, SST-5, and IMDb, covering binary acceptability, five-class sentiment, and binary movie-review sentiment classification.Training sizes are 8,551, 8,544, and 22,500 samples, respectively.
  • Models: The models are BERT-Base with approximately 110M parameters and GPT-2 Large with approximately 762M parameters.BERT-Base has 12 layers and hidden size 768; GPT-2 Large has 36 layers and hidden size 1,280.
  • Baselines: FISGuard is compared with DP-SGD, L2 Regularization, Min-Max, Dropout, and Label Smoothing.These baselines represent gradient perturbation, regularization, adversarial, and output-related defenses.
  • Training configuration: The federated setup uses 30 participating clients, 50 communication rounds, one local update step per round, and local batch size 16.All clients participate in every communication round.
  • Hyperparameters: A public reference set contains 2,000 samples, while defense-specific hyperparameters are swept with other training settings held fixed.DP-SGD noise, L2 coefficients, and Min-Max adversarial weights are varied across specified ranges.

B. Defense Effectiveness and Model Utility

Across 12 configurations, FISGuard substantially weakens ProjRes membership signals while retaining accuracy close to the undefended model. Its fixed public subspace explains the improved coverage of private representations by observable gradients.

  • Privacy protection: FISGuard reduces the member–non-member projection-residual gap and increases distributional overlap relative to the undefended setting.The residual therefore carries weaker membership-discriminative information.
  • Privacy protection: Across 12 configurations, FISGuard’s attack AUC remains close to the random-guessing level of 0.5 in most cases.Its AUC distribution is more concentrated around 0.5 than those of competing defenses.
  • Mechanism: Freezing the input projection maps members and non-members into the same fixed r-dimensional public subspace, increasing coverage by the remaining gradient row space.When that row space fully spans the projected space, the residual difference exploited by ProjRes is eliminated.
  • Model utility: FISGuard accuracy remains close to the undefended baseline and is generally more favorable than that of competing defenses across the 12 configurations.The reported utility results indicate no substantial degradation from the privacy defense.
  • Overall outcome: FISGuard achieves a favorable privacy–utility trade-off across datasets, models, and fine-tuning strategies.The comparison spans three datasets, two LLMs, and Adapter and LoRA strategies.

C. Computation Overhead Analysis

FISGuard adds only limited computational overhead across datasets and fine-tuning strategies, with training time remaining close to the undefended baseline.

  • FISGuard introduces only limited computational overhead across Adapter- and LoRA-based fine-tuning.Under Adapter, it achieves the lowest overall training time among evaluated defenses; under LoRA, runtime remains close to baseline.

VI. SECURITY ANALYSIS

FISGuard changes the protected layer’s observable geometry by projecting representations into a fixed subspace while avoiding a differential-privacy or universal membership-inference guarantee.

  • FISGuard’s security guarantee is specific to the ProjRes projection-residual score and is not a differential-privacy or universal membership-inference guarantee.The analysis concerns how fixed public-subspace reparameterization affects ProjRes.
  • FISGuard reparameterizes the task-specific update as ∆W = UQ, with fixed input projection Q and client-trained output mapping U.The projected representation is z = Qx, and the server observes the output-side gradient rather than an input-side update built directly from x.
  • The protected representation space has dimension r, with r < d, so server-visible updates operate on a lower-dimensional projection of the original representation.Here d is the original representation dimension and r is the projected dimension.

B. Invalidation of the ProjRes Score

When the observed output-side gradient has full rank in the fixed projected space, every candidate’s projection residual is zero, so ProjRes loses membership-discriminating information.

  • rank(G_U) = r makes the row space of G_U equal the entire projected space R^r.Because G_U is in R^{m×r}, full rank r means its row space has dimension r and therefore covers R^r.
  • s_PR(x; G_U) = 0 for every candidate sample, regardless of membership status.Projection onto the full projected space is the identity, leaving zero residual for every projected representation.
  • Adv_PR = 0 and ROC AUC = 1/2 under the standard convention that tied scores receive half credit.Members and non-members receive identical scores, so ProjRes produces no nontrivial ranking.
  • The output-side gradient is G_U = DZ^⊤, where Z contains projected batch representations and D contains backpropagated error signals.This factorization exposes the conditions governing the rank of the observed update.
  • rank(Z) = r and ker(D) ∩ col(Z^⊤) = {0} are sufficient conditions for rank(G_U) = r.The projected representations must span all r directions, while the error signals must not eliminate a nonzero direction in that span.

D. Interpretation and Design Condition

FISGuard relies on the fixed projected representation space being fully covered by the server-observable update span, while experiments report near-random ProjRes AUC with preserved utility and limited overhead.

  • Full coverage requires projected batch representations to span all r fixed-subspace directions and error signals not to eliminate any direction in that span.Because rank(G_U) ≤ min{m, r, B}, full rank requires r ≤ m and r ≤ B; sequence models may instead use effective independent count B_eff.
  • FISGuard constructs a fixed low-dimensional input subspace from independent public data and restricts trainable updates to that subspace.The design aims to reduce member–non-member projection-residual discrepancy while preserving downstream-task information.
  • Across three datasets, two LLMs, and two fine-tuning strategies, FISGuard brings ProjRes AUC near random guessing in most settings while keeping utility close to the undefended model.The evaluation also reports only limited computational overhead and characterizes when the update span fully covers the fixed representation subspace.

APPENDIX A DETAILED RESULTS

The appendix examines how defense-specific hyperparameters affect ProjRes attack AUC and downstream accuracy across datasets, models, and fine-tuning strategies. FISGuard varies the retained representation dimension r, linking privacy behavior to the geometry of its fixed input subspace.

  • Attack AUC: Conventional defenses alter attack effectiveness by changing noise, regularization, adversarial penalties, dropout, or label smoothing across hyperparameter sweeps.These settings change how strongly the original optimization process is perturbed.
  • Attack AUC: FISGuard keeps ProjRes attack AUC close to random guessing in most evaluated configurations as r varies.Changing r alters the dimensionality of the fixed subspace in which observable gradients are formed.
  • Attack AUC: Smaller r increases shared coverage of member and non-member representations, reducing their projection-residual difference.If the gradient row space spans the projected space, both candidate types have identical residuals.
  • Model Accuracy: Figure 6 reports downstream model accuracy for the same defense-specific hyperparameter sweeps across three datasets, two LLMs, and two fine-tuning strategies.For existing defenses, stronger perturbation may suppress leakage while interfering with optimization and reducing task performance.
  • Model Accuracy: FISGuard selects the fixed subspace from the top-r right singular vectors of public representations, retaining dominant pretrained representation directions.Thus, r controls retained representation directions rather than noise or gradient-perturbation magnitude.
Loading 2608.27836v1…