Collaborative Personalized Preference Alignment for LLMs under Data Deficiency

1 INTRODUCTION

• Multi-objective preference alignment tailors outputs to client-specified balances across objectives such as helpfulness, harmlessness, and humor.The introduction frames this as a way to tailor LLM responses to personal preferences.Source: §1 · p. 1
• Personalized federated learning can share an initialization for few-shot adaptation, but heterogeneous preferences and competing objectives create conflicts across clients and within each client.Scarce client-specific feedback also makes training an effective personalized aligner from scratch difficult.Source: §1 · pp. 1–2
• APO learns an initialization for each client cluster, then refines it with adaptation-aware learning so clients can adapt it using few local examples.The paper reports conditional suboptimality bounds for single-step collaborative updates and stochastic adaptation to unseen preferences.Source: §1 · p. 2
• Bottleneck-adjustment clustering reduces inter-client interference, while preference gradient descent with controlled ascent addresses competing objectives within clients.The method groups clients by bottleneck ordering, refines groups with preference-adjustment vectors, and moves within each cluster toward preference-specific Pareto points.Source: §1 · p. 2
• Experiments on UltraFeedback with Llama-3.2-3B-Instruct and Qwen2.5-3B-Instruct demonstrate the superiority of APO over state-of-the-art methods.Source: §1 · p. 3

Personalized Multi-Objective Alignment under Data Scarcity

• Data-scarce methods use synthetic preferences or user demonstrations, while MetaAligner generalizes across unseen objectives but not heterogeneous users; the paper says these approaches rely on abundant centralized or synthetic data.The related-work section states they do not learn personalized models from scarce real-client feedback under preference-specific settings.Source: §2 · p. 3

Collaborative Alignment under Gradient Conflicts

• Federated alignment and personalized FL do not provide few-shot personalization guarantees or model preference-specific Pareto geometry, while EPO handles within-model but not cross-client conflicts.APO combines bottleneck-adjustment clustering, controlled-ascent updates, and adaptation-aware meta-training to address both conflict levels under data deficiency.Source: §2 · p. 3

3 METHODOLOGY

• APO learns a cluster-specific initialization, refines it with adaptation-aware meta-training, and lets a new client adapt the corresponding starting point to its preference.The stated goal is useful starting points for clients with different multi-objective preferences and limited local feedback.Source: §3 · p. 3

3.1 PRELIMINARIES AND PROBLEM SETUP

• Clients have loss vectors and preference vectors, and the weighted Chebyshev objective guides each client toward a preference-specific Pareto solution.Preference Uniformity measures how well the current loss vector aligns with the client preference; it is zero when weighted losses are balanced according to that preference.Source: §3.1 · pp. 3–4

3.2 CONSTRUCTING CLUSTERS: BOTTLENECK-ADJUSTMENT CLUSTERING

• Bottleneck-Adjustment Clustering first groups clients by the ordering of their preference-weighted losses, then hierarchically refines groups by adjustment-vector distance.The refinement aims to reduce within-cluster direction dispersion, which bounds deviation between a client’s local direction and the aggregate under the stated common-basis assumption.Source: §3.2 · pp. 4–5
• When the stated local correction-gain condition holds for every client in a cluster, a shared update decreases every client’s Preference Uniformity.The theorem gives this simultaneous balancing guarantee for a sufficiently small global step size.Source: §3.2 · p. 5

3.3 LEARNING SHARED INITIALIZATIONS: PREFERENCE-AWARE COLLABORATIVE TRAINING

• Initialization learning combines preference balancing with controlled ascent, then switches to a common descent direction once preference balance is achieved.The constraints prevent bottleneck objectives from worsening while allowing bounded degradation of non-bottleneck objectives during balancing descent.Source: §3.3 · pp. 5–6
• A second phase meta-trains the collaborative initialization on S-shot episodes so it can adapt to a client preference within K steps.The episodes are subsamples of each client’s own dataset.Source: §3.3 · p. 6

3.4 THEORETICAL GUARANTEES: FROM COLLABORATIVE CONVERGENCE TO FEW-SHOT PERSONALIZATION

• Under a trajectory hypothesis and an explicit descent-margin condition, the one-local-step shared iteration has horizon-independent asymptotic suboptimality bounded by a fixed radius.The descent-margin condition is PL-type along the actual LP direction and is assumed rather than derived from smoothness, bounded gradients, preference balance, or LP constraints.Source: §3.4 · pp. 6–7
• For a new client assigned to a cluster, the analysis bounds expected stochastic few-shot adaptation suboptimality relative to sufficient-data local training using collaborative-training quantities.The gap vanishes as the cluster tightens only in the strict case where the stated tolerances b_i are zero.Source: §3.4 · p. 7

4.1 EXPERIMENTAL SETUP

• Experiments use Fed-ChatbotPA and UltraFeedback, evaluate Score, HV, and IGD, and compare six baselines using Llama-3.2-3B-Instruct and Qwen2.5-3B-Instruct.The setup describes the helpful-assistant task and reports response scoring with ArmoRM-Llama3-8B-v0.1.Source: §4.1 · p. 8

4.2 MAIN RESULTS

• APO achieves the highest Score on 23 of 24 clusters, highest or tied-highest HV on 22, and lowest or tied-lowest IGD on 21 after adaptation with 20 support samples.The paper reports higher HV and lower IGD as evidence of broader trade-off coverage and closer coverage of the empirical Pareto front.Source: §4.2 · p. 8
• Across all four model–dataset settings, APO improves average Score, HV, and IGD over FedAvg, which uses the same collaborative Pareto-set initialization phase.Average results are consistent across both datasets and base models; the sole non-strict improvement is tied IGD for Llama on Fed-ChatbotPA versus the strongest baseline.Source: §4.2 · p. 9

4.3 ABLATION STUDY

• Average gain remains positive at every evaluated communication round on both datasets.Figure 3 encodes APO’s normalized weighted-score gain over FedAvg, in percentage points, for UltraFeedback and Fed-ChatbotPA with Llama-3.2-3B-Instruct.Source: §4.3 · p. 9, Figure 3 · p. 9
• BAC raises the overall average score from 0.70 to 0.72 and improves the average on six of eight clusters, though HC performs better on C0 and C2.The ablation compares BAC with hierarchical clustering on raw preference vectors; BAC groups clients using bottleneck and adjustment compatibility.Source: §4.3 · p. 9, Table 2 · p. 9

5 CONCLUSION

• APO combines Bottleneck-Adjustment Clustering, preference gradient descent with controlled ascent, and adaptation-aware meta-training to address inter-client and intra-client conflicts.Source: §5 · p. 10
• On Fed-ChatbotPA and UltraFeedback with two 3B backbones, APO improves preference-weighted scores and generally provides better Pareto-front coverage using 20 local examples.Source: §5 · p. 10

• The authors report using generative AI tools for tasks including proof-writing assistance, implementation, translation, dataset cleanup, analysis, and figure or code editing, and say they reviewed the assisted work.Source: AI USE STATEMENT · p. 10
• The authors state that they identify no ethical concerns associated with the work.Source: ETHICS STATEMENT · p. 10
• The paper says its source code will be made public upon acceptance and that the appendix contains proofs, dataset descriptions, and data-processing details.Source: REPRODUCIBILITY STATEMENT · p. 10

• Each Fed-ChatbotPA entry contains a prompt with preference instructions, a chosen response, and a rejected response for one preference dimension.Source: Figure 4 · p. 14
• The example structure includes preference weights.Source: Figure 4 · p. 14
• Figure 4 illustrates an example from the Fed-ChatbotPA dataset (Ye et al., 2024a).Source: Figure 4 · p. 14

• Fed-ChatbotPA uses 6,550 training entities across 131 clients, each receiving 20 support and 30 query entities.Source: B DATASET SETUP · p. 14
• Held-out test sets contain 700 Fed-ChatbotPA entities for 10 clients and 350 UltraFeedback entities for 5 clients; test clients adapt with 20 examples and use the remaining 50 for evaluation.Source: B DATASET SETUP · p. 14
• UltraFeedback uses 4,750 training entities across 95 clients, each receiving 20 support and 30 query entities.Source: B DATASET SETUP · p. 14

• Table 3 covers training hyperparameters for Fed-ChatbotPA and UltraFeedback across Llama-3.2-3B and Qwen2.5-3B.Source: C TRAINING PARAMETERS · p. 14

Phase 1: Basic initializer training

• Phase 1 computes client loss and gradient information, solves Equation 4 for balancing weights, and aggregates client updates with FedAvg to produce wI.The local direction uses balancing descent when preference uniformity βi > 0 and uniform descent when βi = 0.Source: D ALGORITHM · pp. 14–15

Phase 2: Adaptation-aware meta-training

• Phase 2 meta-trains wI on S-shot episodes with Equation 5, returning wII as an initializer intended to reach a client preference within K adaptation steps.Source: D ALGORITHM · pp. 14–15
• Phase 2 uses the same clients, local data, objectives, and server aggregation as Phase 1, differing only in its balancing-LP inner update, K steps, and meta learning rate ηmeta.Source: D ALGORITHM · p. 14

Convergence and adaptation analysis

• The convergence analysis covers a one-local-step idealization of Phase 1 and extends to Phase 2 under the same inner-step idealization; adaptation analysis starts from arbitrary w0.Source: D ALGORITHM · pp. 14–15

Training-free and data-free baselines

• Base, RIC, and Rewarded Soup represent alternatives that use no client-local gradient adaptation: zero-shot prompting, inference-time conditioning, and plug-and-play expert mixing, respectively.Base uses no local samples; Rewarded Soup uses one expert per objective.Source: E BASELINE · p. 15

Few-shot baselines

• FSPO uses synthetic-preference supervised fine-tuning before clients adapt from scarce comparisons with a per-user IPO update.A subset of local comparisons is placed in the prompt and the remaining hold-out pairs are used for updating.Source: E BASELINE · p. 16
• DITTO personalizes from scarce demonstrations through online imitation learning and DPO-style updates on self-generated pairs, without local dispreferred labels.Source: E BASELINE · p. 16

Collaborative baseline

• FedAvg uses the same S-shot, K-step adaptation data as APO, so their comparison isolates the effect of APO’s adaptation-aware meta-trained initializer.Source: E BASELINE · p. 16

Preference-weighted normalized score

• Preference-weighted normalized score is the primary metric for comparing scores across dimensions.Source: F METRICS · p. 16

Nondominated fronts

• The empirical reference front is the nondominated set from the union of all methods, with cluster and preference indexing specified for the evaluation.Source: F METRICS · p. 17

Hypervolume (HV)

• Hypervolume measures dominated objective-space volume relative to the origin; higher values reward fronts with strong objective scores and broader trade-off coverage.Source: F METRICS · p. 17

Inverted generational distance (IGD)

• IGD averages each reference-front point’s distance to its nearest method solution; lower values indicate closer coverage of the empirical reference front.Source: F METRICS · p. 17

G MORE DETAILED RELATED WORK

• Existing multi-objective alignment methods vary in where they inject trade-offs, but generally assume abundant centrally curated preference data and serve generic users.The discussion covers prompt-conditioned, parameter-space, decoding-time, and preference-conditioned approaches.Source: G MORE DETAILED RELATED WORK · p. 17
• Few-shot personalization methods use user variables, prototypes, synthetic preferences, or demonstrations, but do not combine real feedback across clients while modeling multi-objective Pareto structure.FSPO relies on synthetic preferences; DITTO adapts locally from demonstrations.Source: G MORE DETAILED RELATED WORK · pp. 17–18
• Federated alignment methods primarily target population-level collaboration, while personalized federated methods generally use single-objective utility and omit preference-specific Pareto geometry.APO instead clusters by preference bottlenecks and adjustment vectors and bounds initialization distance and the post-adaptation gap.Source: G MORE DETAILED RELATED WORK · p. 18
• Common multi-objective optimization methods seek Pareto-stationary directions without selecting a client-specific trade-off; EPO models a preference but is formulated for one learner.APO extends preference-aware optimization to clients with heterogeneous preferences.Source: G MORE DETAILED RELATED WORK · p. 18
• APO combines bottleneck-adjustment clustering with controlled ascent to address client-level and within-client gradient conflicts, extending preference-aware Pareto optimization to collaborative personalization.The related-work discussion contrasts this with population-level methods and single-learner Pareto optimization.Source: G MORE DETAILED RELATED WORK · pp. 18–19

H.1 ASSUMPTIONS

• The analysis assumes bounded gradients and Lipschitz gradients in the parameter region visited during training.The text motivates boundedness through LoRA, weight decay, and gradient clipping, and describes smoothness for LoRA-parameterized models with bounded activations.Source: §H.1 · p. 19

H.2.1 DECOMPOSING HCn VIA PREFERENCE ADJUSTMENT EQUATIONS

• The quantitative preference-spread bound is unavailable when clients have different local gradient matrices or select different optimal LP bases; then direction deviation must be measured independently.The analysis does not claim clustering lowers heterogeneity throughout the training trajectory.Source: §H.2.1 · pp. 19, 21, 22
• Within a cluster, direction heterogeneity comes from client-dependent adjustment vectors in active LP constraints; if no such rows are active, the directions coincide and heterogeneity is zero.The objective vector can affect which basis is optimal, but not the multiplier value once the basis is fixed.Source: §H.2.1 · p. 20
• With common objective functions, direction dispersion equals the C-weighted dispersion of LP multipliers, making adjustment-vector proximity a surrogate for grouping clients by multiplier similarity.This identity does not require continuity of the LP solution map; the quantitative adjustment-vector bound additionally requires a common non-degenerate basis.Source: §H.2.1 · pp. 19, 21
• For fixed positive losses with the stated normalization floor, the adjustment vector varies Lipschitz-continuously with preference, so tighter preference clusters have smaller meta-update variance under the proposition’s assumptions.The bound’s constant depends on the loss and normalization quantities and worsens when a client’s normalizer is close to zero.Source: §H.2.1 · pp. 21–22
• Under common objective functions and a common non-degenerate optimal basis, cluster direction heterogeneity is bounded by a constant times the within-cluster adjustment-vector variance.The bound applies at the point where the common-basis assumption holds; otherwise heterogeneity is treated as a measured quantity.Source: §H.2.1 · pp. 19, 20, 22

H.3 COLLABORATIVE PREFERENCE UNIFORMITY DESCENT UNDER PREFERENCE HETEROGENEITY

• Cluster preference direction heterogeneity measures the weighted dispersion of clients’ local directions around their aggregated direction.With equal aggregation weights, it is the ordinary sample variance of local directions.Source: §H.3 · p. 23
• A shared direction with positive inner product against every client’s balancing direction is a simultaneous balancing direction, and a sufficiently small shared update decreases every client’s Preference Uniformity.The theorem assumes bounded and Lipschitz gradients and specifies local-step and direction-deviation quantities.Source: §H.3 · p. 23
• The proof links the shared-direction guarantee to a positive inner product with each client’s balancing direction and shows that a sufficiently small step does not increase that client’s non-uniformity.The conclusion is established simultaneously for all clients in the cluster.Source: §H.3 · pp. 23–25

H.4 COLLABORATIVE INITIAL ALIGNER CONVERGENCE TO CLUSTER-LOCAL OPTIMA.

• Under an explicit LP descent-margin condition and a trajectory hypothesis that keeps iterates in the residual-floor regime, one shared solution converges within a cluster-tightness-controlled range of every client’s Chebyshev optimum.Source: §H.4 · pp. 25, 29
• When all clients have βi = 0, the collaborative aggregate is an approximate simultaneous descent direction; it becomes exact in the homogeneous limit δg = 0.Source: §H.4 · p. 26
• In mixed clusters, the balancing update decreases βi under a range condition, while uniform-descent clients receive only approximate descent with a penalty from the balancing block.Source: §H.4 · pp. 27–28
• A shared point cannot lie on every distinct preference ray, so balancing need not eliminate all clients’ preference uniformity; the section quantifies this residual floor under regularity assumptions.Source: §H.4 · p. 28
• The H.19 descent-margin condition is a genuine PL-type restriction: it excludes non-minimizing stationary points and can fail near regular convex nonsmooth minima, where the required tolerance may not vanish with step size.Source: §H.4 · p. 29
• A horizon-independent guarantee relies on Assumption H.19, which is not implied by the basic assumptions; its sufficient strict form requires local PL curvature and near-active direction comparability.The curvature condition is strong and differs from quadratic growth; direction comparability concerns the actual LP direction and is not guaranteed by the LP constraints.Source: §H.4 · p. 30
• When a client’s range condition fails, its residual preference heterogeneity is bounded at that iterate; this does not show that the whole cluster enters or stays in the low-heterogeneity region.In mixed clusters, cross-block pull widens the cone around the preference ray, while the later convergence theorems assume the region holds along the trajectory.Source: §H.4 · p. 31
• The convergence analysis covers only the synchronous shared iteration with one local LP step per communication round (τ = 1), not the multi-local-step algorithm or K > 1 inner steps.Extending the result would require analyzing clients’ averaged local directions and a descent-margin condition; the LP solution map may change discontinuously with the shared point.Source: §H.4 · p. 32
• The shared update’s exact one-step loss bounds have no o(η) remainder when the update segment stays within the region where the assumptions hold.Every weighted loss, including inactive objectives, is bounded by the same right-hand side.Source: §H.4 · p. 32
• Under the stated trajectory hypotheses and η ≤ 1/cd, each client’s Chebyshev suboptimality contracts toward ϵ0,i = κH,i ∆bar_i + κη,iη + bi/cd, without guaranteeing iterate convergence.The bound is approached to any positive tolerance, but the closed band need not be entered in finite time.Source: §H.4 · p. 34
• The admissible sets are nested up to per-step slack and decrease toward a fixed limiting cluster band.Source: §H.4 · pp. 34–35
• Under the descent-margin and trajectory conditions, each client’s objective gap contracts geometrically and eventually lies within a horizon-independent band of radius ϵ0,i = κH,i ∆bar_i + κη,i η + bi/cd.The asymptotic guarantee means iterates eventually enter the band plus any ζ > 0; it does not require a limit point to exist.Source: §H.4 · pp. 35–36

H.4 COLLABORATIVE INITIAL ALIGNER CONVERGENCE TO CLUSTER-LOCAL OPTIMA

• If a limit point exists and satisfies the local quadratic-growth condition, the objective-space guarantee also yields a parameter-space bound.Local quadratic growth is an explicit assumption and is used only for this conversion.Source: §H.4 · p. 35, §H.5 · p. 38
• The guarantees are conditional: the coordinatewise upper bound does not imply proximity to the Pareto front, and the descent-margin assumption is not implied by smoothness or the update constraints.The theorem covers synchronous updates with one local step per round, not the multi-step algorithm verbatim.Source: §H.4 · pp. 36–37
• Under the theorem’s idealizations, Phase 2 continues the shared iteration and inherits its objective-band guarantees with a two-phase radius.The analysis assumes one inner step per round and population gradients; it does not quantify the gap from the S-shot implementation.Source: §H.4 · pp. 36–37

H.5 THE INITIALIZER APPROXIMATES EACH CLIENT’S INDIVIDUAL OPTIMUM

• A new client can inherit the cluster guarantee when it shares the cluster’s objectives and its preference lies within the cluster’s preference spread.The result does not cover clients with different loss functions.Source: §H.5 · pp. 37–38
• The initializer’s objective-space error decomposes into a collaborative training gap, a preference gap δλ∥l(w∗)∥∞, and an optimum spread δλL∗.The training gap is bounded by the cluster-average convergence-band radius; the decomposition is horizon-independent under the stated assumptions.Source: §H.5 · p. 39

H.6 DATA-SCARCE ADAPTATION ON THE INITIAL ALIGNER: THE δ0 GAP

• O(σ2/S) is the statistical floor in the expected gap between idealized data-scarce adaptation and data-rich local training.The other component depends on the starting-point suboptimality and the number of local adaptation steps.Source: §H.6 · pp. 40, 44
• The analysis idealizes fresh independent batches of S samples at every step, so its guarantees are not for reusing a fixed total budget of S samples.The experiments use S = 20 fixed local examples; analyzing that setting would require optimization analysis on the fixed dataset and a generalization bound.Source: §H.6 · pp. 40–41
• The bounds concern an idealized preconditioned stochastic process, not the AdamW optimizer used in experiments.The stated differences include momentum, preconditioner measurability, and decoupled weight decay.Source: §H.6 · p. 41
• The initialization gap is explicitly bounded under the stated cluster conditions, but its tolerance term has no proved vanishing rate unless bi = 0.The bound also depends on cluster tightness, step sizes, and the descent tolerance.Source: §H.6 · pp. 43–44
• Each idealized local step contracts the expected population gap by 1−αµPLp−, reducing the initialization component geometrically over K steps.The initialization bound depends on the starting gap ∆0(w0).Source: §H.6 · pp. 40, 44