PaperlayerView exact revision on arXiv

Source-linked AI summary

Personalized Federated Direct Preference Optimization via LoRA Adapters

Zhanhong Jiang, Soumik Sarkar

arXiv:2610.12885v1cs.LG

TL;DR

FedDPO’s shared-policy assumption is a poor fit for heterogeneous client preferences. pFedDPO keeps client-specific LoRA adapters local while synchronizing a shared base model, and the paper reports lower gradient norms than vanilla FedDPO with competitive preference accuracy.

  • Problem

    FedDPO analysis assumes clients converge to a common global policy, although real clients can have heterogeneous preferences that should not be averaged away.

    Source: Introduction · p. 1, Discussion on pFedDPO · p. 7

  • Method

    pFedDPO trains a private LoRA adapter for each client atop a shared base model that is synchronized each round.

    Source: Introduction · p. 1, Personalized FedDPO (pFedDPO) · p. 2, Conclusion · p. 4

  • Results

    pFedDPO consistently attains a lower gradient norm than vanilla FedDPO while keeping preference accuracy competitive across clients.

    Source: Conclusion · p. 4

  • Takeaways & Limitations

    The design keeps personalization local while the shared base captures shared structure, without increasing communication compared with standard FedDPO.

    Source: Conclusion · p. 4, Discussion on pFedDPO · p. 7

    Experiments use DistilGPT-2, 120 randomly partitioned pairs per agent, single-seed ablations, and training-pair accuracy; larger models, domain partitions, held-out evaluation, and multiple seeds remain future work.

    Source: Limitations · p. 12

Abstract

from arXiv · show

Federated Direct Preference Optimization (FedDPO) enables privacy-preserving alignment of language models across distributed agents without centralizing sensitive preference data. However, existing FedDPO analysis assumes all clients converge to a single shared policy, which is fundamentally at odds with the heterogeneous preference distributions present in practice. We propose Personalized FedDPO (pFedDPO), in which each client maintains a small private LoRA adapter atop a shared global base model. In pFedDPO, the base model is synchronized via FedAvg each round, while LoRA adapters remain local and capture client-specific preference patterns. We prove that this decomposition replaces the gradient diversity $\kappa^2$ in the FedDPO bound with a residual heterogeneity $\kappa^2_{\mathrm{eff}}$, which is no larger than $\kappa^2$ when adapters absorb client-specific directions, at the cost of an additive adapter-bias term that vanishes as adapters converge. Experiments on preference datasets show that pFedDPO consistently attains a lower gradient norm than vanilla FedDPO while remaining competitive in training loss and per-client reward accuracy.

Introduction

FedDPO’s shared-policy assumption and irreducible gradient-diversity floor do not fit clients with genuinely different preferences. pFedDPO keeps client-specific LoRA adapters private while synchronizing a shared base, replacing the bound’s heterogeneity term with a residual one plus adapter bias.

Introduction

  • The method targets FedDPO’s shared-policy assumption, which is unrealistic when client preferences differ and leaves an irreducible gradient-diversity floor.The floor cannot be reduced by communicating more or training longer at a fixed step size.Source: Introduction · p. 1

Personalized FedDPO (pFedDPO)

  • pFedDPO keeps each client’s LoRA adapter private while synchronizing the shared base model through FedAvg each round.Clients update both base and adapter locally, but upload only the base-model update; communication remains the same dimension as vanilla FedDPO.Source: Personalized FedDPO (pFedDPO) · p. 2
  • κ^2_eff is no larger than κ^2 to first order under a shared-subspace condition.The residual measures heterogeneity in the base gradients after client adapters absorb directions they can express.Source: Personalized FedDPO (pFedDPO) · p. 2
  • The analysis assumes approximate adapter stationarity, bounding the bias in each client’s base-gradient estimate by a term proportional to ε_ϕ.The proof applies the FedDPO convergence analysis to the adapter-adjusted client objectives, with the bias added after telescoping.Source: Personalized FedDPO (pFedDPO) · p. 2
  • The convergence analysis replaces gradient diversity κ^2 with residual heterogeneity κ^2_eff, while adding an adapter-bias term Cε_ϕ.The resulting bound is tighter than vanilla FedDPO when the heterogeneity reduction exceeds Cε_ϕ.Source: Personalized FedDPO (pFedDPO) · p. 2
  • The expected bound trade-off with LoRA rank is non-monotone: higher rank may further reduce κ^2_eff but can increase adapter bias when adapters are harder to converge.The paper describes an optimal rank as balancing these effects.Source: Personalized FedDPO (pFedDPO) · p. 2

Experiments

On SHP, pFedDPO has a lower gradient norm than vanilla FedDPO with comparable loss and accuracy. Ablations show that rank and client participation affect the measured outcomes.

Experiments

  • pFedDPO attains a lower gradient norm than vanilla FedDPO with comparable loss and accuracy on SHP.Figure 2 compares the baselines; the experiment uses five agents and 80 communication rounds.Source: Experiments · pp. 2–3, Figure 2 · p. 3
  • By round 80, all five agents reach approximately 0.75–0.85 preference accuracy under pFedDPO, with a mean near 0.80.The roughly 10-percentage-point spread is consistent with heterogeneous preferences across agents.Source: Figure 3 · p. 3, Experiments · p. 3
  • At round 80, rank r = 4 reaches approximately 0.82 reward accuracy, while r = 8 has the lowest final gradient norm, approximately 0.02.Accuracy is broadly similar across ranks, and its relationship with rank is non-monotone.Source: Figure 4 · p. 3, Experiments · p. 4
  • Empirical κ² on pFedDPO’s base-model gradients falls from approximately 1500 at initialization to 150–200 by round 70, a reported reduction of roughly 85–90%.The decline is not monotone; adapters are zero-initialized, so the starting value matches vanilla FedDPO’s κ².Source: Figure 5 · p. 4, Experiments · p. 4
  • For both methods, participation ranks S = 5 above S = 3 above S = 1; pFedDPO’s lower gradient-norm floor is clearest at S = 5 and S = 3.At S = 1, both methods struggle, with gradient norms near 1.0.Source: Figure 6 · p. 4, Experiments · p. 4

Related Work

Prior work studies centralized DPO and distributed preference optimization, but the cited federated DPO approaches do not combine personalization with an explicit convergence analysis. The paper situates pFedDPO against federated fine-tuning and personalized LoRA work.

Related Work

  • Centralized DPO and its extensions focus on a single-model setting, while FedDPO and DecDPO provide convergence guarantees for distributed DPO (Jiang 2026).Earlier distributed preference work primarily studies learning and, more recently, agentic AI systems (Saadati et al. 2024, 2025; Esfandiari et al. 2021; Rillo et al. 2026).Source: Related Work · p. 4
  • The cited federated DPO approaches either lack personalization or introduce it empirically without bounding its convergence effect (Zhu et al. 2026).The former treats clients symmetrically; the latter is described as lacking a bound on how personalization improves convergence.Source: Related Work · p. 4
  • Personalized federated LoRA has been studied outside preference optimization, where trajectory-level DPO introduces additional variance and heterogeneity requiring explicit analysis (Yi et al. 2023; Qi et al. 2024).Source: Related Work · p. 4

Conclusion

The conclusion presents pFedDPO as personalized federated DPO with private LoRA adapters, lower gradient norms than vanilla FedDPO, and competitive accuracy. The related-work comparison emphasizes its explicit convergence bound and the trade-offs against other federated approaches.

Conclusion

  • pFedDPO lowers gradient norms relative to vanilla FedDPO while keeping preference accuracy competitive across clients.Its analysis introduces a bias that diminishes over training, and the method preserves personalization without increasing communication over standard FedDPO.Source: Conclusion · p. 4

Additional Related Work

  • Unlike single-policy FedDPO, pFedDPO targets the irreducible κ² floor in its bound (Jiang 2026).Source: Additional Related Work · p. 7
  • Spadea and Seneviratne (2025) compare KTO and DPO empirically, but do not address personalization or provide convergence analysis.Source: Additional Related Work · p. 7
  • FedPDPO shares LoRA updates and reports up to 4.80% average accuracy improvement, whereas pFedDPO keeps adapters private and gives an explicit convergence bound relating personalization to κ².pFedDPO uses full d-dimensional base updates, the same volume as vanilla FedDPO.Source: Additional Related Work · p. 7
  • Compared with FedPDPO, pFedDPO keeps personalization parameters on-device at higher upload volume; compared with Spadea and Seneviratne, it personalizes to client preferences.The paper identifies its distinguishing feature as a rate-explicit convergence bound.Source: Additional Related Work · p. 7

Discussion on pFedDPO

pFedDPO keeps client LoRA adapters private while synchronizing a shared base model. This separates globally transferable alignment from client-specific preferences, which the paper argues should not be averaged away.

Discussion on pFedDPO

  • Unlike methods that periodically aggregate LoRA adapters, pFedDPO keeps them private and synchronizes only the shared base model.The design reflects the paper’s view that user values, style, and alignment objectives are genuine differences rather than noise.Source: Discussion on pFedDPO · p. 7
  • Under heterogeneous preferences, shared adapters can mix task-level adaptation with client-specific information, and averaging may dilute personalized signals or create preference interference.The paper notes that DPO’s pairwise-preference gradients are highly sensitive to local preference distributions.Source: Discussion on pFedDPO · p. 7
  • The shared base captures globally transferable alignment knowledge, while private adapters capture client-specific residual preferences, preventing aggregation of incompatible preference-specific parameters.Source: Discussion on pFedDPO · p. 7
  • Keeping adapters local provides a stronger privacy boundary and reduces effective client drift, while communication remains unchanged from standard FedDPO because only base updates are sent.The paper describes this as a trade-off among global coordination, local personalization, privacy, and robustness to preference heterogeneity.Source: Discussion on pFedDPO · p. 7

Hyperparameters

The section points readers to Table 2 for the empirical hyperparameters.

Hyperparameters

  • Table 2 lists the hyperparameters used in the empirical evaluation.The passage does not specify the hyperparameter values.Source: Hyperparameters · p. 7

Proof of Theorem 1

The proof follows the structure of Jiang’s Theorem 5.1 and highlights modifications due to LoRA personalization. It bounds adapter-related error and absorbs its drift contribution into the bias term under a step-size condition.

Proof of Theorem 1

  • The proof follows Jiang’s Theorem 5.1 and highlights the modification introduced by LoRA personalization.The provided passages do not specify further details of the inherited proof structure.Source: Proof of Theorem 1 · p. 7
  • Cross-Lipschitzness and the PL condition bound the adapter-related gradient error by (L_θϕ/μ_ϕ)^2 ε_ϕ.The derivation identifies this contribution with Cε_ϕ.Source: Proof of Theorem 1 · p. 7
  • The adapter-bias drift contributes O(L^2η^2E^2)ε_ϕ and is absorbed into Cε_ϕ when η ≤ 1/(8LE).The drift argument adapts the cited lemmas by replacing ∇L_i with ∇F_i.Source: Proof of Theorem 1 · pp. 7–8

Intuition

The intuition is that adapters can absorb some client-gradient heterogeneity within adapted parameter blocks. The reduction depends on adapter stationarity, which modules are adapted, and how client adapter subspaces align.

Intuition

  • At adapter stationarity, the base gradient has no component in the adapter tangent space.The loss depends on the effective weight W_ℓ + sB_i,ℓA_i,ℓ, linking the base and adapter gradients.Source: Setup · p. 8, An Exact Identity at Adapter… · p. 9
  • The residual heterogeneity is bounded below by κ_N − δ, so heterogeneity in non-adapted parameters remains.The bound uses the component on θ_N, where the complementary projector is the identity.Source: From Stationarity to Reduced… · p. 9, An Exact Identity at Adapter… · p. 9
  • Shared adapter subspaces eliminate the misalignment term; weak coupling between tangent and orthogonal spaces makes δ second order in the adapter shift.With misaligned subspaces, the misalignment term is paid explicitly, and κ²_eff ≤ κ² is not guaranteed.Source: From Stationarity to Reduced… · p. 9
  • Adapting only c_attn cannot remove heterogeneity in MLP, embedding, or LayerNorm parameters, regardless of rank.Increasing rank enlarges the adapter tangent space, and κ²_eff is nondecreasing in rank for nested adapter subspaces.Source: Role of the LoRA Rank r · p. 10
  • Heterogeneity absorption can saturate once adapter rank reaches the numerical ranks of the mean-zero heterogeneity blocks.Beyond that point, increasing rank removes no further heterogeneity.Source: Role of the LoRA Rank r · p. 10
  • On hh-rlhf, gradient norm decreases monotonically through rank 16, while larger adapters may increase the adapter-bias term if they are harder to bring near stationarity.The difficulty claim is presented as a hypothesis; the rank ablation is described as consistent with the proposed trade-off.Source: Role of the LoRA Rank r · p. 10
  • The proposed low-rank explanation is that concentrated sigmoid weights make client heterogeneity blocks numerically low rank; the effect of β is conjectured and untested.All experiments use β = 0.2.Source: DPO-Specific Structure · p. 10
  • The reduction is largest when heterogeneity lies in adapted modules; when non-adapted heterogeneity dominates, adapting more modules may help more than increasing rank.The text identifies MLP projections as an example of additional modules to adapt.Source: When Is the Reduction Largest? · p. 10

2. The adapted-block heterogeneity lies in the adapters’ tangent spaces, i.e., κ2

Private adapters act as client-specific pre-filters on base gradients, resolving locally the disagreement they can express in adapted blocks. The reduction is conditional on aligned filters and small curvature leakage, and measured effective diversity alone cannot separate adapter effects from ordinary training progress.

2. The adapted-block heterogeneity lies in the adapters’ tangent spaces, i.e., κ2

  • Adapters resolve locally the client disagreement expressible in their tangent spaces; the uploaded remainder includes orthogonal gradients and non-adapted parameters.This mechanism targets the inter-client heterogeneity that leaves an irreducible floor in vanilla FedDPO.Source: Intuition · p. 10
  • The filter removes heterogeneity only in adapted modules and does so cleanly when client filters are roughly aligned and curvature leakage is small.Otherwise heterogeneity can remain in the orthogonal complement.Source: Intuition · p. 10, §2 · p. 10
  • Measured effective gradient diversity on the pFedDPO trajectory cannot by itself separate adapter effects from ordinary training progress.The estimate is subject to adapter bias when adapters are not near stationarity.Source: Intuition · p. 10, §2 · p. 10

Additional Results

On SHP and hh-rlhf, pFedDPO generally lowers gradient norm while keeping loss and per-client accuracy competitive. Additional analyses show that rank, participation, and dataset heterogeneity shape the observed outcomes.

SHP Table

  • A falling κ^2 trajectory alone cannot separate the adapter effect from ordinary training progress.The passage identifies this as a limitation of interpreting the measured trajectory.Source: Additional Results · p. 11

Comparative Study

  • ∼0.2: on hh-rlhf, pFedDPO reached the lowest gradient-norm floor by round 80, versus centralized DPO’s ∼0.6–1.0; vanilla FedDPO descended similarly but settled slightly higher.Centralized DPO fluctuated without stable convergence in this setup.Source: Additional Results · p. 11
  • ∼0.95: by round 80, pFedDPO reached hh-rlhf preference accuracy versus vanilla FedDPO’s ∼0.92 and centralized DPO’s ∼0.80; both federated methods reached loss ∼0.10–0.15.The passage notes hh-rlhf accuracies were higher than SHP’s and suggests pair structure as a plausible factor, not label quality.Source: Additional Results · p. 11

Effect of LoRA Personalization on Client Drift

  • ∼45–55%: hh-rlhf’s empirical κ^2 fell overall from ∼900 to ∼400–500 by round 70, with oscillations; the gradient-norm improvement was smaller than on SHP.Raw κ^2 values are not directly comparable across datasets because gradient scales differ.Source: Additional Results · p. 11

Per-Client Reward Accuracy

  • ∼0.90–0.92: mean hh-rlhf per-client reward accuracy by round 80, with agent values from ∼0.88 to ∼0.97 and a 9–10 percentage-point spread.The experiments use five agents with 120 disjoint pairs each; reward accuracy is measured on each agent’s training pairs.Source: Additional Results · p. 11

LoRA Rank Ablation

  • ∼0.19: rank 16’s final gradient norm, down from ∼0.36 at rank 2; accuracy stays near ∼0.90–0.92 across ranks, and rank 8 remains the default although its gradient-norm benefit has not saturated.The passage reports rank effects on gradient norm exceed those on accuracy, and the best rank appears dataset-dependent.Source: Additional Results · pp. 11–12

SHP Table

  • 0.02: on SHP at round 80, pFedDPO had the lowest gradient norm, while vanilla FedDPO led training loss (0.05) and reward accuracy (0.82); pFedDPO reached 0.06 and 0.80.Table 3 summarizes round-80 SHP results.Source: Additional Results · p. 11, Table 3 · p. 12

Participation Ablation

  • ∼2×: at S = 5, pFedDPO’s hh-rlhf gradient norm (∼0.32) is about half vanilla FedDPO’s (∼0.65); pFedDPO at S = 3 reaches ∼0.25 by round 80.At S = 1, pFedDPO fluctuates around 0.4–0.8 without clear convergence.Source: Additional Results · p. 12

Limitations

The analysis and experiments rely on assumptions and settings that limit how broadly the results can be applied.

Limitations

  • The convergence analysis inherits log-linear softmax policy assumptions; extending the bounds to transformers requires controlling Fisher information spectra.The paper leaves that extension for future work.Source: Limitations · p. 12
  • Theorem 11 assumes adapter-wise µϕ-PL local losses and a unique minimizing low-rank shift; κ^2_eff ≤ κ^2 is established only to first order under a shared-subspace condition.The authors characterize these assumptions as strong for bilinear LoRA parameterizations.Source: Limitations · p. 12
  • The adapter-bias constant C lacks a closed-form expression in E, η, and local curvature, limiting direct use of the bound to select E.Empirical evaluation also uses DistilGPT-2, 120 randomly partitioned pairs per agent, one-seed ablations, and training-pair accuracy.Source: Limitations · p. 12

Ethics Statement

The ethics discussion weighs privacy and preference plurality against risks from information leakage, harmful private adapters, and malicious clients. Its experiments use two open academic preference datasets, and it calls for additional safeguards before high-stakes deployment.

Privacy and data minimization

  • pFedDPO shares base-model updates, not raw preferences or adapters, but those higher-dimensional updates may leak local data through gradient inversion.The paper identifies differential privacy as a natural extension but does not implement it because its interaction with the gradient-diversity bound needs dedicated analysis.Source: Ethics Statement · p. 12

Preference heterogeneity and value plurality

  • Private adapters may preserve minority preferences, but can also encode harmful or biased preferences that global averaging might moderate.The paper recommends adapter-level content monitoring or periodic auditing.Source: Ethics Statement · p. 12

Adversarial clients and Byzantine robustness

  • Like FedDPO, pFedDPO is vulnerable to malicious base-model updates, while private adapters may conceal intent from the server.The paper recommends considering robust aggregation in adversarial settings.Source: Ethics Statement · p. 15

Scope of application

  • Experiments use two open academic preference datasets, HH-RLHF and SHP, for research validation; high-stakes deployment requires additional safeguards.The paper names human oversight, regular safety-benchmark evaluation, and ways for clients to audit or contest optimization.Source: Ethics Statement · p. 15

Related papers

Loading 2610.12885v1…