PaperlayerView exact revision on arXiv

Source-linked AI summary

Balancing Privacy, Utility, and Safety in LLM Alignment through Preference Optimization

Dishu Yang, Jingjing Liu, Jize Li

arXiv:2608.30141v1cs.CRcs.LG

TL;DR

Preference data can expose memorization and membership risks, motivating tests of whether privacy-oriented data mixing changes these signals during DPO. The paper introduces P3M, which varies synthetic privacy-preference data while holding helpfulness and harmlessness data fixed, and evaluates two Gemma settings with canary and membership-inference diagnostics. Under the tested conditions, privacy-aware configurations show lower canary proxy values across both settings and lower aggregate 2B attack metrics, but source-level reductions and harmlessness effects are not uniform.

  • Problem

    Preference data may be vulnerable to membership inference and memorization-related exposure, but how privacy-oriented preference-data amounts affect these signals within fixed DPO settings is limited.

  • Method

    P3M varies the amount of synthetic privacy-preference data in DPO while keeping helpfulness and harmlessness data fixed, evaluating two Gemma settings with canary and PREMIA-based diagnostics.

  • Results

    Across both model settings, privacy-aware configurations have lower mean canary proxy values; in the mixed-source 2B evaluation, AUROC is 0.596 to 0.629 and AUPRC is 0.541 to 0.575 versus Baseline values of 0.804 and 0.790.

  • Takeaways & Limitations

    P3M is a lightweight empirical protocol for examining privacy-utility-safety trade-offs, with helpfulness broadly stable but harmlessness effects varying by model setting.

  • Takeaways & Limitations

    Metric differences may reflect changed nominal pool size, duplicated privacy rows, and sampled-example variation, because privacy rows are added or repeated rather than substituted.

Abstract

from arXiv · show

Preference optimization is widely used to align large language models with human preferences, but preference-data composition may also influence privacy-relevant memorization. We examine whether adding synthetic privacy-preference pairs to Direct Preference Optimization (DPO) is associated with lower canary-based memorization signals without modifying the objective or introducing a formal privacy mechanism. We propose Privacy-Pressure Preference Mixing (P3M), a data-composition protocol that varies the amount of privacy-preference data while keeping helpfulness and harmlessness preference data fixed. We evaluate a non-privacy Baseline and privacy-mixing ratios of 0.5, 1.0, and 2.0 using Gemma 3 270M-IT across five random seeds and validate the same four conditions using 4-bit-quantized Gemma 2 2B-IT across three seeds. Overall, under the tested conditions, privacy-preference mixing is associated with lower mean canary suffix log-likelihood proxy values across both model settings and lower aggregate membership-inference attack performance relative to the Baseline in the mixed-source 2B evaluation. Specifically, across the privacy-aware 2B configurations, the mean area under the receiver operating characteristic curve (AUROC) ranges from 0.596 to 0.629, and the mean area under the precision-recall curve (AUPRC) ranges from 0.541 to 0.575, compared with 0.804 and 0.790, respectively, for the Baseline. However, the reduction in membership distinguishability does not hold uniformly across data sources. Moreover, the relationship between the privacy ratio and harmlessness preference accuracy varies by model setting, whereas helpfulness preference accuracy remains broadly stable. These findings suggest that P3M should be viewed as a lightweight empirical protocol for examining privacy-utility-safety trade-offs rather than as a formal privacy guarantee or a defense against extraction attacks.

I. INTRODUCTION

The paper asks whether varying privacy-oriented preference data within otherwise fixed DPO experiments changes helpfulness, harmlessness, and memorization-related signals. It introduces P3M and evaluates these outcomes across two Gemma settings using complementary empirical diagnostics.

  • I. INTRODUCTION: P3M varies privacy pressure by adding synthetic privacy-preference pairs while keeping helpfulness and harmlessness preference data fixed.Chosen responses omit synthetic private identifiers, while rejected responses reveal them.
  • I. INTRODUCTION: The evaluation compares a non-privacy Baseline with privacy-mixing ratios 0.5, 1.0, and 2.0 across Gemma 3 270M-IT and 4-bit Gemma 2 2B-IT.The experiments use five and three random seeds, respectively.
  • I. INTRODUCTION: The study jointly measures helpfulness preference accuracy, harmlessness preference accuracy, and a canary suffix log-likelihood proxy, with PREMIA added for the 2B validation.The canary proxy and PREMIA provide complementary empirical signals of memorization and membership distinguishability, not formal privacy guarantees.
  • I. INTRODUCTION: Across both model settings, privacy-aware configurations have lower mean canary proxy values than their corresponding Baselines.In the 2B validation, aggregate AUROC and AUPRC are also lower on mixed-source attack sets, but source-stratified reductions are not uniform.
  • I. INTRODUCTION: The paper reports broadly stable helpfulness accuracy, while the relationship between privacy mixing and harmlessness accuracy differs between the 270M and 2B settings.These findings motivate jointly evaluating privacy, utility, and safety outcomes.
  • I. INTRODUCTION: The contributions include P3M, two controlled model evaluations, and aggregate and source-stratified PREMIA-based membership-inference analysis.The aggregate reduction relative to the Baseline does not hold uniformly across helpfulness, harmlessness, and privacy-preference tuples.

II. RELATED WORK

Related work frames preference-based alignment as learning from human comparisons for tasks including summarization, instruction following, and helpful-and-harmless assistants. It also identifies memorization and data-exposure risks that extend into preference-data alignment.

  • II. RELATED WORK: Preference-based learning uses human comparisons to learn reward functions and has been applied to summarization, instruction following, and helpful-and-harmless assistant training.
  • II. RELATED WORK: Language models may memorize rare, unique, or duplicated training sequences, creating a risk of data exposure.Canary testing quantifies exposure, while extraction attacks can reproduce verbatim training examples, including personally identifiable information.

C. Privacy-Aware Alignment and Safety Trade-offs

The paper situates P3M among privacy-aware alignment and resource-efficient adaptation methods. It treats privacy pressure as a DPO data-composition variable while keeping core training components fixed within each model experiment.

  • C. Privacy-Aware Alignment and Safety Trade-offs: Privacy-aware alignment work includes differential privacy, privacy-preserving synthetic instructions, and federated RLHF approaches.
  • C. Privacy-Aware Alignment and Safety Trade-offs: LoRA and QLoRA reduce computational requirements by training low-rank adapters over frozen base models, with QLoRA using a 4-bit-quantized base model.These methods serve as computational tools rather than privacy mechanisms.
  • C. Privacy-Aware Alignment and Safety Trade-offs: P3M varies privacy-preference data across mixing ratios while fixing the base model, DPO objective, LoRA configuration, and helpfulness and harmlessness pools.Adapters are evaluated using helpfulness and harmlessness preference accuracy and a canary suffix log-likelihood proxy, with larger-model validation extending the evaluation.

B. Preference Data Construction

The preference-data construction combines fixed helpfulness and harmlessness pairs with synthetic privacy-preference pairs. Privacy pressure is changed by sampling half, all, or duplicated privacy rows while leaving the non-privacy pools fixed.

  • B. Preference Data Construction: The fixed non-privacy data comprise helpfulness and harmlessness preference pairs from Anthropic HH-RLHF.These represent utility and safety, respectively, in the DPO training mixture.
  • B. Preference Data Construction: The synthetic privacy pool uses artificial identifiers, with chosen responses withholding each identifier and rejected responses disclosing it.The base pool contains 100 unique identifiers across low-, medium-, and high-frequency tiers, yielding 538 rows.
  • B. Preference Data Construction: The privacy-aware training set combines helpfulness, harmlessness, and synthetic privacy-preference datasets.Privacy pressure is introduced through dataset composition rather than a new objective.
  • B. Preference Data Construction: At ratios 0.5, 1.0, and 2.0, privacy-preference rows are sampled at half frequency, included fully, or duplicated, respectively.The base model, optimizer, DPO objective, and LoRA configuration remain fixed, while nominal pool size changes.

E. Evaluation Protocol

The evaluation measures utility, safety, and privacy-related signals using held-out preference accuracy, a canary suffix log-likelihood proxy, and PREMIA-based membership inference.

  • Evaluation metrics: Utility and safety are measured by held-out helpfulness and harmlessness preference accuracy, respectively.A prediction is correct when the chosen completion receives the higher length-normalized conditional log-probability.
  • Evaluation metrics: Lower canary suffix log-likelihood proxy values indicate a weaker measured memorization-related signal.The proxy averages conditional log-probabilities of hidden canary suffixes across frequency tiers and is not a formal privacy guarantee or extraction test.
  • Membership inference: The 2B validation uses a length-normalized log-domain PREMIA adaptation to score preference tuples.The tuple contains a prompt, chosen response, and rejected response, with scores based on target/reference probability ratios.
  • Membership inference: The tuple-level attack score compares DPO-aligned and frozen pre-DPO mean response-token log-probabilities.Higher scores indicate stronger evidence that the complete preference tuple was used during alignment.
  • Membership inference: Membership is assigned when a unique tuple is sampled at least once during 100 DPO optimization steps.Unsampled tuples from the training pool serve as non-member candidates, and balanced attack sets report AUROC and AUPRC.

IV. EXPERIMENTS AND RESULTS

The experiments compare a Baseline with three privacy-mixing ratios across two Gemma settings, jointly evaluating preference accuracy, canary memorization signals, and membership inference in the 2B validation.

  • Experimental design: The Baseline and privacy ratios 0.5, 1.0, and 2.0 are evaluated on Gemma 3 270M-IT across five seeds and 4-bit Gemma 2 2B-IT across three seeds.Both settings measure helpfulness accuracy, harmlessness accuracy, and the canary suffix log-likelihood proxy; the 2B validation additionally reports PREMIA AUROC and AUPRC.

A. Experimental Setup

The setup evaluates four data-composition conditions on primary and larger-model experiments, using matched preference-accuracy, canary, and membership-inference measures.

  • Experimental setup: The four configurations are evaluated on Gemma 3 270M-IT across five seeds and 4-bit Gemma 2 2B-IT across three seeds.The 2B attack sets contain 83 to 100 member tuples and equal numbers of non-member tuples per run, with source composition matched where possible.
  • Experimental setup: Table 1 summarizes the nominal training-pool composition.The composition is organized around the experimental training conditions and preference-data pools.
  • Evaluation criteria: Higher preference accuracy indicates better performance, while lower canary proxy values and values closer to 0.5 for AUROC and AUPRC indicate weaker measured privacy-risk signals.These metrics are interpreted as empirical signals rather than formal privacy guarantees.

C. Primary Gemma 3 270M-IT Results

Across the primary 270M experiment and 2B validation, privacy-aware configurations show lower canary proxy values while helpfulness remains broadly stable. Membership-inference reductions are aggregate, source-dependent, and accompanied by model-dependent harmlessness patterns.

  • Gemma 3 270M-IT: All privacy-aware 270M configurations have lower canary proxy values than the Baseline (-7.952), with ratio 2.0 lowest (-10.644).Mean helpfulness accuracy remains 0.579 to 0.588, while harmlessness accuracy decreases from 0.461 to 0.427.
  • Gemma 2 2B-IT: All privacy-aware 2B configurations yield lower canary proxy values than the Baseline (-7.348), with ratio 2.0 lowest (-31.905).The ratio 2.0 configuration also has the greatest variability across seeds.
  • Gemma 2 2B-IT: 2B helpfulness accuracy remains broadly stable at 0.598 to 0.616, while harmlessness does not decline monotonically with increasing privacy ratio.Harmlessness accuracy rises to 0.489 ± 0.073 at ratio 2.0 after being modestly lower at ratios 0.5 and 1.0.
  • Membership inference: Privacy-aware 2B configurations reduce aggregate AUROC to 0.596–0.629 and AUPRC to 0.541–0.575 versus Baseline values of 0.804 and 0.790.These results indicate weaker aggregate membership distinguishability, but neither metric changes monotonically with privacy ratio.
  • Membership inference: Source-specific attack subsets contain 17 to 55 members and non-members per class, and the 2B validation uses only three seeds.The authors characterize these source-level results as descriptive diagnostics; the Baseline has no privacy-preference tuples for a corresponding source-specific result.
  • Membership inference: Source-stratified results do not show uniform membership-distinguishability reductions: ratios 0.5 and 1.0 generally improve over Baseline, whereas ratio 2.0 is comparable to or higher.Aggregate results therefore reflect mixture-level rather than uniform source-level distinguishability.

V. DISCUSSION

Across both model settings, privacy-preference mixing is associated with lower canary proxy values, while helpfulness remains stable and harmlessness and membership-inference patterns vary by setting and source.

  • V. DISCUSSION: P3M is a data-composition protocol for examining privacy-utility-safety trade-offs, not a formal privacy defense or uniformly beneficial intervention.The discussion limits the conclusion to the tested empirical conditions.

A. Interpreting the Privacy Mixing Ratio

The privacy ratio changes both privacy-preference data volume and nominal pool size, so observed differences cannot be attributed solely to composition. Results also vary across model settings and evaluation sources, requiring joint interpretation of privacy, helpfulness, and harmlessness outcomes.

  • Interpreting the Privacy Mixing Ratio: Higher privacy ratios lower mean canary proxy values in both models, but PREMIA performance is nonmonotonic and source-specific reductions are inconsistent.The monotonic harmlessness decline seen in the 270M experiment does not recur in the 2B validation.
  • Interpreting the Privacy Mixing Ratio: The 2B experiment is a larger-model validation, not a controlled scaling study, because the settings differ in model generation, parameter scale, and quantization.Generalization to other model families and scales remains future work.
  • Interpreting the Privacy Mixing Ratio: The privacy ratio changes nominal pool size because privacy-preference rows are added or repeated rather than substituted for non-privacy rows.Metric differences may therefore also reflect duplicated rows, pool size, and examples sampled across runs.
  • Interpreting the Privacy Mixing Ratio: Canary and PREMIA results are empirical diagnostics rather than formal privacy guarantees or evidence of extraction resistance.PREMIA membership uses the realized update stream, and varying attack-set composition can affect aggregate comparisons.
  • Interpreting the Privacy Mixing Ratio: Synthetic canaries avoid real personal data but do not represent leakage involving realistic identifiers, contextual clues, or partial-information combinations.More representative privacy-sensitive formats and extraction prompts should be evaluated with safeguards.

VI. CONCLUSION

P3M varies synthetic privacy-preference data while holding helpfulness and harmlessness pools fixed within each model experiment. Under the tested conditions, privacy-aware configurations show lower canary-based signals and lower aggregate mixed-source membership-attack metrics, but trade-offs and limitations prevent treating P3M as a formal privacy defense.

  • VI. CONCLUSION: P3M varies synthetic privacy-preference data while keeping helpfulness and harmlessness pools fixed within each model experiment.
  • VI. CONCLUSION: Privacy-aware configurations lower mean canary proxy values across model settings and lower aggregate AUROC and AUPRC in the 2B mixed-source evaluation.Source-stratified membership distinguishability does not decline uniformly, and harmlessness trends differ between model settings.
  • VI. CONCLUSION: Privacy-mixing configurations should be evaluated jointly across privacy, utility, and safety outcomes rather than selected using a single objective.
  • VI. CONCLUSION: P3M is a lightweight empirical protocol, not a formal privacy guarantee or evidence of resistance to extraction attacks.Future work should broaden controls, model families, privacy-sensitive formats, membership-inference protocols, and extraction evaluations.
Loading 2608.30141v1…