PaperlayerView exact revision on arXiv

Source-linked AI summary

Who Should We Listen to More? Welfare-Aware Preference Acquisition for Pluralistic Alignment

Ruomeng Ding, Tianwei Gao, Lianrui Geng, Mohit Bansal, Zhun Deng

arXiv:2610.19035v1cs.LGcs.AIcs.CL

TL;DR

Pluralistic alignment faces the problem of allocating limited human feedback when AI simulators represent people’s preferences with varying accuracy. The paper proposes WAPA, which estimates welfare gains to select queries, and reports improved supervision and policy preference prediction, including substantial recovery of the predictive-utility gap with a 20% label budget.

  • Problem

    Limited human feedback must be allocated across people whose preferences AI simulators represent with varying accuracy, and useful feedback alone does not specify how gains should be valued across recipients.

    Source: §1 · pp. 1–2

  • Method

    WAPA uses auxiliary reference models to estimate query gains and persona utilities, then selects queries by marginal welfare gain under an objective incorporating current utility and application-designated priority.

    Source: §1 · pp. 2–3

  • Results

    Across Roleplay and OpinionQA, WAPA improves supervision and enables allocation tradeoffs; with 20% human labels, it recovers 75–86% of the average predictive utility gap between fully AI-labeled and ground-truth-labeled policies.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The welfare objective makes allocation priorities explicit, accounting for both poorly represented preferences and application-designated priorities.

    Source: §1 · p. 3, §5 · p. 12

    The exact optimality guarantee covers the estimated acquisition welfare objective under fixed predictions, not downstream policy welfare; guarantees for iterative acquisition adapting to labels and updated policies remain open.

    Source: B Limitations · p. 21

Abstract

from arXiv · show

Pluralistic alignment needs to accommodate diverse preferences under limited human feedback. Using AI simulators to label preference comparisons reduces annotation costs, but variation in simulation accuracy across individuals can leave some people's preferences poorly represented in training. We study how to allocate human feedback across people to construct training data for pluralistic alignment, and how this allocation affects the trained policy. To make this allocation explicit and controllable, we define a social welfare objective over individual supervision utilities that measure label quality. Each query's marginal welfare gain depends on its expected supervision improvement, the recipient's current utility, and application-designated priority. Welfare-Aware Preference Acquisition (WAPA) uses auxiliary reference models to estimate query gains and individual utilities, then selects queries by estimated marginal welfare gain while updating recipient utilities. We prove exact optimality of the acquisition algorithm for the estimated welfare objective under a fixed query budget and model predictions. Experiments on six Roleplay collections and four OpinionQA regions demonstrate improved supervision and controllable allocation tradeoffs. Policy training on four Roleplay collections improves preference prediction for both seen and unseen individuals. When prioritizing a designated group and poorly represented individuals, acquiring human labels for 20% of comparisons recovers 75-86% of the average predictive utility gap between policies trained entirely on AI-generated labels and entirely on ground-truth labels.

1 Introduction

Pluralistic alignment must allocate limited human feedback across people whose preferences AI simulators represent with varying accuracy. The paper formulates this allocation as a welfare objective and introduces WAPA, reporting improved supervision and policy preference prediction under targeted allocation.

1 Introduction

  • Because similar average supervision gains can benefit different people, the paper asks how acquisition should balance improvements across recipients and whether allocation choices translate to downstream alignment.The setup uses AI simulators to predict individual preferences over response pairs, with human feedback allocated to selected comparisons under a limited budget.Source: §1 · pp. 1–2
  • The welfare objective values supervision gains according to recipients’ current supervision utilities and fixed application-designated priorities.These priorities need not align: an application-priority persona may already be well represented, while a poorly represented persona may not belong to the priority group.Source: §1 · p. 2
  • With human labels for 20% of comparisons, WAPA recovers 75–86% of the average predictive utility gap between policies trained entirely on AI-generated labels and entirely on ground-truth labels.The experiments span six Roleplay collections and four OpinionQA regions; policy training on four Roleplay collections improves preference prediction for seen and unseen personas.Source: §1 · p. 3
  • The paper formulates human-feedback allocation for pluralistic alignment as a social welfare objective that accounts for poorly represented preferences and application-designated priorities.It presents this as a welfare-based formulation of fair feedback allocation for pluralistic-alignment training data.Source: §1 · pp. 2–3
  • WAPA estimates query gains and persona utilities with auxiliary reference models, then selects queries by marginal welfare gain while updating recipient utilities.Its greedy acquisition algorithm exactly maximizes the estimated welfare objective under a fixed query budget and fixed model predictions.Source: §1 · p. 3
  • Experiments characterize how welfare priorities shape allocation and supervision quality, while policy training improves preference prediction over Random and welfare-matched Entropy at matched feedback budgets.The policy-training comparison is reported for Roleplay collections.Source: §1 · p. 3

2 Problem Setup

The setup treats feedback acquisition as choosing persona–prompt comparisons from a shared pool under a fixed label budget. It measures each persona’s supervision quality and aggregates it using weights and a curvature parameter that encode allocation priorities.

2 Problem Setup

  • A query reveals one comparison’s true preference label; the resulting dataset retains all comparisons, using observed labels for queried items and base-model soft pseudo-labels elsewhere.The comparison pool pairs each persona with common prompts and two fixed candidate responses.Source: §2 · p. 4
  • Persona supervision utility measures probabilistic label quality using a strictly proper scoring rule, with Brier loss as the concrete implementation.Queried comparisons contribute zero loss because their labels are known; unqueried comparisons retain the target model’s predictions.Source: §2 · p. 4
  • The welfare curvature parameter ρ controls sensitivity to low supervision utility: concavity for ρ < 1 favors gains for lower-utility personas, and smaller ρ strengthens that priority.The subgroup weights separately encode application-specific priority; increasing γ > 1 raises the weights of target-subgroup members.Source: §2 · p. 4
  • For ρ = 1, suitable positive normalized weights can support allocations meeting jointly feasible subgroup requirements, whereas negative ρ shapes the tradeoff between target priority and fairness for lower-utility personas.Setting (γ,ρ) = (1,1) recovers average supervision utility maximization.Source: §2 · pp. 4–5

3 Method

WAPA estimates the welfare consequences of human-feedback queries and greedily allocates a fixed budget using those estimates. The method connects the selected feedback to policy training and analyzes what its welfare objective and acquisition guarantee establish.

3.1 Welfare-Aware Preference Acquisition (WAPA)

  • WAPA estimates persona utilities and welfare from cross-model predictions, then selects the query with the largest estimated welfare gain and updates only that recipient’s estimated utility.It uses precomputed model predictions throughout acquisition, without model retraining.Source: §3.1 · pp. 5–6
  • The welfare-gain expansion reflects both target-subgroup priority and greater weight for personas with lower current estimated supervision utility when ρ < 1.Its first-order terms represent estimated utility gain, target-subgroup priority, and low-utility priority.Source: §3.1 · pp. 5–6

3.2 From Preference Acquisition to Pluralistic Alignment

  • Selected human labels and remaining AI supervision jointly train the policy with DPO; the welfare priorities direct observed feedback toward poorly captured or higher-priority preferences.A human-label-based refinement yields similar downstream Mean predictive utility with direct pseudo-label use and improves Tail utility (Appendix G.16).Source: §3.2 · p. 6

3.3 Theoretical Analysis

  • Suitable positive subgroup weights can make a welfare maximizer meet jointly feasible subgroup requirements, while unweighted mean-utility maximization can leave a subgroup’s requirements unmet.The theorem gives a two-subgroup instance where unweighted mean maximization leaves one group at average utility δ despite a feasible requirement u1 = 1.Source: §3.3 · p. 7
  • Choosing negative ρ adds a tradeoff between target-subgroup priority and fairness for personas disadvantaged in supervision quality.Source: §3.3 · p. 7
  • For a unique feasible max–min allocation, sufficiently small ρ makes that allocation the unique maximizer of both weighted and unweighted welfare.This result holds for fixed ε > 0 and positive normalized weights.Source: §3.3 · p. 7
  • WAPA’s greedy output exactly maximizes the estimated welfare objective for fixed model estimates, including when scores are zero or tied.The guarantee requires neither calibrated reference predictions nor independence among labels or reference models.Source: §3.3 · p. 7

4 Experiments

The experiments evaluate WAPA’s acquisition quality, welfare-parameter tradeoffs, and downstream policy effects across preference datasets and persona splits. WAPA leads budgeted acquisition methods, while welfare settings shift who benefits and the acquired labels improve downstream preference prediction.

4.1 How effective is WAPA at preference acquisition under different welfare specifications?

  • At a 20% query budget, WAPA has the highest Mean, Tail, and Target acquisition utility across four main datasets and both welfare settings, beating the strongest baselines by 2.86–3.24, 3.18–4.63, and 1.85–3.64 percentage points, respectively.It also leads the evaluated methods at every budget from 1% to 40%.Source: §4.1 · pp. 8–9, Table 1 · p. 9
  • Figure 2 tracks acquisition utility by query budget under (2,−5), evaluating methods at 1% increments and reporting the initial-to-oracle utility gap recovered at 20%.The oracle uses full-label access.Source: Figure 2 · p. 9

4.2 How do welfare parameters shape allocation and utility tradeoffs at the acquisition stage?

  • At a 20% budget, switching from (1,1) to (2,−5) raises WAPA’s Target utility by 3.72–4.09% while reducing Mean by 0.11–0.20%; Tail also falls in most datasets.Figure 3 compares parameter effects and outcome tradeoffs relative to (1,1), with a random 10% Target.Source: §4.2 · p. 10, Figure 3 · p. 10
  • At ρ = 1, increasing γ from 1 to 2 raises Target utility by 9.01–10.06 percentage points while reducing Mean and Tail, and directs over a third of the query budget to the roughly 10% Target group.At γ = 2, lowering ρ from 1 to −5 raises Mean by 0.41–0.62 points and Tail by 1.16–1.54 points relative to (2,1), while reducing Target utility.Source: §4.2 · p. 10

4.3 Does WAPA improve downstream alignment?

  • With 20% queried labels, WAPA recovers 74.84–85.74% of the Mean improvement from all pseudo-labels to all ground-truth labels across four datasets and both persona splits.BAL-PM recovers 49.72–59.05% and matched Entropy 40.23–49.38% of the same reference gap; WAPA also yields higher Mean and Tail utility than the compared methods across both welfare settings.Source: §4.3 · p. 11, Figure 4 · p. 11, Figure 5 · p. 11
  • Under (2,−5), WAPA recovers 71.58–104.71% of the Tail reference gap across both persona splits, and its Target utility exceeds matched Entropy on Seen and similarity-defined Held-out personas.Recovery can exceed 100% because the fully labeled policy is a reference, not a performance bound.Source: §4.3 · p. 11
  • In an exploratory evaluation of 128 rankings over 96 persona–prompt instances, WAPA’s mean rank is better than Random and Entropy in both regions and close to Full-GT’s, though Full-GT receives more first-place rankings.Thirteen blinded raters assessed response suitability; WAPA’s overall mean rank is 2.298 versus 2.309 for Full-GT.Source: §4.3 · p. 12, Table 2 · p. 12

5 Conclusion and Discussion

The paper frames pluralistic preference acquisition as allocating a shared human-feedback budget across personas, with welfare priorities made explicit and policy training used to assess the resulting supervision. Its discussion distinguishes this task from adjacent acquisition, personalization, and alignment settings.

5 Conclusion and Discussion

  • WAPA allocates queries by marginal welfare gain, and its greedy rule exactly optimizes the estimated objective under a fixed budget and fixed model predictions.The conclusion presents this guarantee alongside experiments in which training on the acquired data improves prediction for seen and unseen personas.Source: §5 · p. 12

A Related Work

  • The related-work overview groups prior work around feedback acquisition, personalization, pluralistic alignment, human–AI allocation, and welfare-sensitive acquisition.Table 3 summarizes the four research directions reviewed in the section.Source: A Related Work · p. 17

A.1 Active acquisition for preference modeling and policy alignment

  • Prior acquisition methods target feedback for learning: APL uses generation entropy and implicit preference certainty [Muldrew et al., 2024], while ActiveDPO uses policy implicit-reward gradients [Lin et al., 2026].The paper’s focus is how to distribute supervision gains from a shared budget across personas.Source: §A.1 · pp. 17–18

A.2 Personalized and pluralistic alignment

  • The paper’s acquisition constructs supervision for persona-conditioned policy training and complements welfare objectives applied during policy optimization.Its batch is selected before labels are observed.Source: §A.2 · p. 18

A.3 Human–AI feedback allocation

  • HyPER predicts reward-model performance for candidate human–AI annotation mixtures, but its experiments leave transfer to DPO or reinforcement-learning policy training open.The paper evaluates acquired supervision through DPO policy training.Source: §A.3 · p. 19

A.4 Social welfare and fair acquisition

  • Unlike welfare objectives that specify how a policy should serve different parties, this acquisition objective allocates alignment feedback using persona-level supervision quality.The paper states its guarantee for the estimated acquisition objective under fixed predictions.Source: §A.4 · p. 19

A.5 Positioning our acquisition problem

  • The paper combines feedback acquisition for policy training with explicit allocation priorities across personas, using AI feedback for unqueried comparisons.Downstream policy training tests the value of that supervision for alignment.Source: §A.5 · p. 20

B Limitations

The limitations bound what the evaluation and theoretical guarantees establish, while the appendix specifies WAPA’s frozen-prediction implementation and its computational cost.

B Limitations

  • Free-response evaluation is exploratory, and the theoretical guarantees do not establish that maximizing acquisition welfare maximizes downstream policy welfare.The risk result also assumes independent sampling of personas and prompts, conditionally independent labels, acquisition independent of pool labels, and bounded empirical fitting error.Source: B Limitations · pp. 20–21

C.1 WAPA algorithmic details

  • For each persona, WAPA retains only its highest-scoring unqueried comparison because gain increases with score and selection changes only that persona’s risk.Other personas’ retained gains therefore remain valid after a selection.Source: §C.1 · p. 21
  • With predictions frozen, total time is O(NMT + NM log(M + 1) + BN), excluding the cost of generating model predictions.The stated bound is polynomial in N, M, and T because B ≤ NM.Source: §C.1 · p. 21

C.2 Proofs of the Allocation-Priority Results

The proofs show how welfare weights can satisfy jointly feasible subgroup requirements that unweighted mean utility may miss. They also establish bottom-tail protections from negative curvature and characterize a limiting max–min allocation.

C.2 Proofs of the Allocation-Priority Results

  • At ρ = 1, at least one weighted-welfare maximizer meeting subgroup requirements also maximizes mean utility among allocations that meet them.The supporting weights may depend on the instance and requirements.Source: §C.2 · pp. 23–24
  • The proof constructs a mean-utility optimum by selecting mandatory feasible group prefixes, then filling the remaining budget with the largest unselected losses.Within each group, selecting the largest losses for a fixed query count maximizes its average utility without changing other groups’ utilities.Source: §C.2 · pp. 23–24
  • The result establishes existence for positive subgroup-constant weights, not feasibility for every optimizer or for every jointly feasible requirement under the one-parameter target-weight family.The supporting weights may depend on the instance and requirements.Source: §C.2 · p. 24
  • With uniform weights and ρ < 0, maximizing welfare also maximizes an explicit lower-bound certificate on bottom-α utility.The certificate increases with the welfare objective.Source: §C.2 · pp. 24–25
  • When the max–min allocation is unique, sufficiently negative ρ makes it the unique optimizer for both weighted and unweighted welfare.This holds for any fixed positive weights.Source: §C.2 · pp. 24–26

C.3 Proof of WAPA Optimality

The proof shows that WAPA’s greedy allocation exactly maximizes its estimated welfare objective for each fixed query budget. It does so by reducing the choice to prefix allocations and selecting the largest available welfare increments.

C.3 Proof of WAPA Optimality

  • WAPA’s output is exactly optimal for the estimated welfare objective at every integer query budget, including cases with zero scores and ties.The guarantee requires neither calibrated reference predictions nor independence among labels or reference models.Source: §C.3 · p. 26
  • For fixed persona query counts, selecting that persona’s highest-scoring comparisons maximizes its estimated utility and cannot decrease welfare.This follows because welfare is increasing in each persona’s utility.Source: §C.3 · p. 26
  • Each persona’s successive welfare increments are nonincreasing, so the greedy algorithm can select the largest available increment while preserving prefix-feasible allocations.The proof orders each persona’s query scores and shows that increasing utility makes later increments no larger.Source: §C.3 · p. 27
  • The proof covers ties and both empty and full query budgets; its guarantee concerns the estimated objective rather than true supervision quality.Within-persona ties can be ordered consistently with the run under consideration.Source: §C.3 · p. 27

C.4 Proof of Acquisition-to-Alignment Transfer

This section gives a conditional bound connecting WAPA’s completed-supervision quality and training fit to preference prediction on fresh personas and prompts. Under its sampling and acquisition assumptions, higher mean supervision utility tightens the population-risk upper bound.

C.4 Proof of Acquisition-to-Alignment Transfer

  • Here, persona and prompt inputs define fresh evaluation draws, while the risk bound uses completed targets and empirical fitting error.The fitting error differs from the training objective by a term independent of the model parameters.Source: §C.4 · pp. 27–28
  • Under the stated assumptions, the trained predictor’s population Brier risk is bounded by remaining supervision error, fitting tolerance, and complexity and sampling terms.The bound holds with probability at least 1 − δ.Source: §C.4 · pp. 28–29
  • The guarantee assumes independently sampled personas and prompts, conditionally independent labels, acquisition independent of pool labels, and a predictor fitted within ε_fit.The predictor class is fixed before sampling, and the training fit must meet the stated tolerance.Source: §C.4 · p. 28
  • Holding other bound terms fixed, higher mean supervision utility yields a tighter upper bound on downstream preference prediction error.The utility measures expected completed-supervision quality on the sampled pool.Source: §C.4 · pp. 28–29

D Supplementary Details on Methods

The supplementary methods explain how reference predictions estimate supervision utility and query gains, how acquired labels and AI supervision form soft-target DPO training data, and how an optional refinement changes unqueried probabilities without changing acquisition.

D.1 Supervision Utility and Estimated Query Gains

  • Expected Brier utility is uniquely maximized by predicting the true preference probability, and WAPA estimates query gains from reference predictions and label uncertainty.The reference mean suffices for the score: ensembles with the same mean yield the same WAPA score, and gains require only marginal label probabilities.Source: §D.1 · pp. 29–30
  • A query can improve supervision even when target and reference predictions agree, because observing the label resolves uncertainty about that comparison.The passage’s example has target, reference, and true probabilities all equal to 1/2 while estimated query gain remains positive.Source: §D.1 · p. 30

D.2 Preference Completion and DPO Objective

  • Soft-target DPO uses observed labels for queried pairs and retains the target model’s orientation and probability for unqueried pairs.The DPO reference policy is a frozen copy of the initialization and is distinct from the models providing acquisition evidence.Source: §D.2 · p. 30

D.3 Post-acquisition Preference Refinement

  • The optional post-acquisition refinement changes predicted probabilities for unqueried comparisons while leaving the query budget and acquired set unchanged.Selected probabilities can replace unqueried targets in soft-target DPO; acquired labels remain unchanged.Source: §D.3 · p. 31
  • Refinement smooths model predictions and adds label residuals using reference and acquired-label weights.The localization procedure splits unqueried personas into two halves, using the opposite half as each persona’s reference pool.Source: §D.3 · p. 31
  • A residual-bridge procedure selects a refinement strength using acquired labels and retains the raw predictor when no candidate meets its criterion.The bridge assumes a shared conditional residual relationship across queried and unqueried comparisons.Source: §D.3 · p. 31

E Datasets

The experiments use Roleplay for acquisition and downstream alignment and OpinionQA for acquisition-only evaluation, then compare welfare-aware and welfare-agnostic query strategies under shared acquisition settings. The setup distinguishes label-acquisition budgets from other computational costs and includes a welfare-optimal oracle for comparison.

E.1 Roleplay

  • Roleplay supplies synthetic personas and model-labeled preference pairs across six collections, with four main collections used for downstream training.The main-text collections are Africa, Asia, North America, and Europe; downstream training retains the full acquisition support.Source: §E.1 · pp. 31–32
  • The Roleplay splits evaluate seen personas on new comparisons and held-out personas excluded from acquisition and training.The held-out target cohort is selected by description similarity to acquisition-target members, which is not a validated measure of behavioral preference equivalence.Source: §E.1 · p. 32

E.2 OpinionQA

  • OpinionQA extends acquisition evaluation to four U.S. regional subsets comprising 3,130 respondents and 93,900 question–respondent pairs.Each respondent answers the same 30 questions, and results are reported in the appendix.Source: §E.2 · p. 33

F Detailed Experimental Setup

  • Reference probabilities come from hosted services, while the target model supplies target probabilities; equal query budgets compare label efficiency rather than total computational cost.WAPA additionally uses reference predictions, and workload accounting is stage-specific because some runtimes were not consistently recorded.Source: §F.1 · p. 33, §F.2 · pp. 33–34

F.3 Baselines and Welfare Weighting

  • The acquisition setup compares equal-weight mean welfare with target-priority, inequality-sensitive welfare at a 20% main budget.The budget is B = floor(0.2NM) for each Roleplay collection.Source: §F.3 · p. 34
  • Entropy and Reference Disagreement use welfare-weighted first-order variants, while WAPA allocates using reference-expected Brier utility.All methods are evaluated using the same true-label utilities; WAPA’s score uses the target probability and equal-weight reference mean.Source: §F.3 · p. 34
  • APL and BAL-PM provide welfare-agnostic acquisition baselines, with selections shared across welfare settings.Both are evaluated over the same 0%–100% budget grid in 1% increments.Source: §F.3 · p. 35
  • Welfare-optimal Oracle ranks true within-persona losses and applies the exact marginal allocator, yielding maximum realized scalar welfare for a fixed budget.For equal-weight welfare, its allocation is the global top-B true losses; under target-priority, inequality-sensitive welfare, it can differ.Source: §F.3 · p. 35

F.4 Evaluation Protocol and Metrics

The evaluation protocol distinguishes acquisition-stage supervision utility from trained-policy predictive utility and reports both across population mean, lower-tail, and designated-group summaries. Supplementary experiments extend acquisition comparisons to OpinionQA regions and additional Roleplay collections, while describing downstream training and free-response audit scope.

F.4 Evaluation Protocol and Metrics

  • Brier-based predictive utility evaluates probability quality, while acquisition utility uses observed labels for queried comparisons and AI-generated probabilities elsewhere.Downstream utility instead scores trained-policy preference predictions on test comparisons; Mean weights personas equally, Tail averages the lowest 10%, and Target averages a fixed designated group.Source: §F.4 · pp. 35–37

F.5 Downstream Alignment Training

  • Downstream experiments use soft-label DPO on four Roleplay datasets, comparing budgeted methods with all-pseudo-label and all-ground-truth references.Budgeted arms replace queried 20% of target probabilities with acquisition labels and retain soft probabilities elsewhere; the reported runs use three epochs and final adapters without validation-based checkpoint selection.Source: §F.5 · pp. 37–38

F.6 Free-Response Generation

  • The free-response audit uses generated outputs from multiple backends, but per-response backend identities were not recorded.Source: §F.6 · p. 38

G Supplementary Experiment Results

  • Supplementary experiments extend the main acquisition comparison to four OpinionQA regions and two additional Roleplay collections.The OpinionQA and additional-Roleplay results compare 20%-budget acquisition utility under matched welfare specifications.Source: §G.1 · p. 38, §G.2 · p. 39

G.1 OpinionQA: Acquisition across Four US Regions

  • Under equal-weight welfare, WAPA exceeds the strongest budgeted baseline across four OpinionQA regions by 3.49–3.57% in Mean, 3.09–3.96% in Tail, and 1.61–4.10% in Target.Under target-priority, inequality-sensitive welfare, Mean and Tail advantages are 2.95–3.25% and 4.40–6.23%; in South, Reference Disagreement exceeds WAPA in Target by 0.97%.Source: §G.1 · pp. 38–39, Table S3 · p. 39

G.2 Roleplay-South America and Roleplay-Heterogeneous

  • At 20% budget, WAPA exceeds the strongest budgeted baseline in Mean, Tail, and Target on both additional Roleplay collections under both welfare specifications.Across comparisons, gains range from 2.74–3.02% in Mean, 2.43–3.95% in Tail, and 2.69–3.14% in Target.Source: §G.2 · pp. 39–40, Table S4 · p. 39

G.3 Equal-Weight Mean Welfare Budget Curves

  • Figure S1 reports equal-weight-welfare acquisition curves for the four main Roleplay collections across Mean, Tail, and Target metrics and query budgets.The plotted methods are evaluated at 1% budget increments, and APL and BAL-PM retain the same welfare-agnostic selections across welfare specifications.Source: §G.3 · p. 40, Figure S1 · p. 40

G.4 Ablation of the Acquisition Score

The supplementary studies test WAPA’s score, allocator, reference composition, designated subgroup, welfare shift, and downstream outcomes across backbones and lower-tail population fractions. WAPA retains acquisition or downstream advantages in the reported settings, with tradeoffs across welfare outcomes and different reference-gap recovery across backbones.

G.4 Ablation of the Acquisition Score

  • Across four datasets, adding reference-mean uncertainty raises Target utility by 0.72–1.03 points under (2,−5), with little change in Mean utility.Discrepancy alone and the full score differ by at most 0.070 percentage points in Mean gain; uncertainty alone has lower Mean and Tail utility than the full score across both settings.Source: §G.4 · pp. 40–41, Table S5 · p. 41

G.5 Sensitivity to the Allocation Rule

  • Replacing exact WAPA marginal allocation with a first-order approximation produces at least 99.9879% query-set overlap and utility differences below 0.00213%.The controlled comparison uses four main Roleplay collections, a 20% budget, and (γ,ρ) = (2,−5).Source: §G.5 · p. 41, Table S6 · p. 41

G.6 Sensitivity to Reference-Model Composition

  • Every tested single-reference and leave-one-out variant exceeds the strongest corresponding original baseline in Mean, Tail, and Target across four datasets and both welfare settings.The full ensemble improves Mean over every single-reference variant by 0.128–1.643%, while subsets can achieve higher Tail or Target utility.Source: §G.6 · p. 42, Table S7 · p. 42

G.7 Sensitivity to the Designated Subgroup

  • When the designated group is the 10% with lowest initial reference-estimated supervision utility, (2,−5) raises Target utility by 3.32–3.77% but lowers Mean by 0.14–0.26% and Tail by 0.23–0.63%.The designated membership is fixed before allocation and differs from Tail, which is reranked by final predictive utility.Source: §G.7 · pp. 42–43, Figure S2 · p. 43

G.8 Sensitivity to the Welfare Shift

  • Across every tested shift and dataset, WAPA exceeds the strongest matched baseline on Mean, Tail, and Target utility.Within WAPA, increasing the shift raises Target utility while reducing Mean and Tail in these experiments.Source: Table S8 · p. 44, §G.8 · p. 44

G.9 Absolute Predictive Utility after Downstream Alignment

  • Tables S9 and S10 report absolute predictive utilities for seen and held-out personas, while Figures S3 and S4 show reference-gap recovery under (1,1).These results accompany the recovery plots under (2,−5); the supplied passages describe the tables and figures but do not state their specific outcome values.Source: §G.9 · p. 44, Table S9 · p. 45, Table S10 · p. 45

G.10 Comparison across Backbones

  • On Llama, WAPA leads all four budgeted baselines in Mean, Tail, and Target for both utility and accuracy; its Tail utility advantage over Entropy is 0.89–1.05 percentage points.The comparison covers North America and Africa, while the reported Tail advantage spans their four dataset–split combinations.Source: §G.10 · p. 46, Table S11 · p. 47, Table S12 · p. 47

G.11 Downstream Lower-Tail Utility Curves

  • Figure S5 shows broadly positive lower-tail utility gains over Random and matched Entropy under both welfare specifications, extending the comparison beyond the single Tail summary.The 10% and 100% population-fraction positions correspond to Tail and Mean, respectively.Source: §G.11 · pp. 48–49

G.12 Downstream Sensitivity to Welfare Parameters

These appendix analyses examine how WAPA’s downstream effects vary with welfare settings and proxy-label completion, including specified hard-label DPO and free-response cases, and document preference-collection and alignment-training templates. The free-response evaluation uses acquired dataset labels alongside hard pseudo-labels; Full-GT denotes the policy trained with all ground-truth labels, not a reference answer.

G.12 Downstream Sensitivity to Welfare Parameters

  • The effects of welfare parameters vary: increasing Target priority raises Target utility at ρ = −5 but lowers it at ρ = 1, and lowering ρ improves Tail only at γ = 1.At ρ = −5, increasing γ from 1 to 2 raises Seen Target utility by 0.0489 points on North America and 0.0002 on Africa; at ρ = 1, it lowers utility by 0.0999 and 0.0984 points. At γ = 1, lowering ρ from 1 to −5 increases Mean by 0.0156 and 0.0351 points and Tail by 0.1024 and 0.0858; at γ = 2, Tail falls by 0.0153 and 0.0210 points.Source: §G.12 · pp. 49–50, Table S16 · p. 50

G.13 Sensitivity to the Imputation Source

  • With reference-based probability completion, WAPA retains the highest Mean and Target among methods across four datasets and both settings, while target completion gives WAPA higher Mean and Target but lower Tail.Reference probabilities improve all three metrics before acquisition; WAPA’s fixed query sets are compared across completion sources.Source: §G.13 · p. 51, Table S17 · p. 51
  • Reference majority voting improves WAPA’s Mean, Tail, and Target label accuracy over target labels across four datasets and both settings, while WAPA retains Mean and Target advantages over matched baselines.The audit uses completed-label accuracy with Tail reranked by per-persona accuracy; these metrics differ from probability-based Brier scoring.Source: §G.13 · p. 51, Table S18 · p. 52

G.14 Downstream Alignment with Hard Proxy Labels

  • Under hard-label DPO, WAPA improves Mean, Tail, and Target utility over Random and matched Entropy across four datasets and both splits under both specifications.Under (2,−5), it recovers 68.11–81.85% of the Mean reference gap and 65.01–94.93% of the Tail reference gap; matched Entropy recovers 43.17–53.88% and 30.14–45.96%, respectively, and Random recovers 29.47–43.96% and 23.55–39.37%.Source: §G.14 · p. 52, Figure S6 · p. 52, Table S19 · p. 53

G.15 Hard Accuracy for Acquisition and Downstream Alignment

  • WAPA improves Mean acquisition accuracy over the strongest budgeted baseline across four datasets, and improves Mean downstream accuracy across both splits and settings.Acquisition gains are 0.44–0.91 percentage points under (1,1) and 0.28–0.80 under (2,−5); downstream gains are 0.51–1.29 points. Under (2,−5), WAPA leads all reported acquisition and hard-label downstream accuracy summaries; with soft-label DPO it leads or ties all Seen summaries, with stated Held-out exceptions.Source: §G.15 · p. 54, Table S23 · p. 55, Table S24 · p. 56

G.16 Sensitivity to Proxy-Label Refinement

  • PPCI reduces unqueried-probability Brier loss but changes downstream Mean utility little; Tail gains, while Target changes are mixed.Brier loss falls by 11.04–12.03% across eight conditions, with α = 0 throughout; downstream Mean changes by less than 0.10 percentage points, Tail rises by 0.25–0.58, and Target ranges from −0.12 to +0.23 points.Source: §G.16 · p. 56, Table S26 · p. 57, Table S27 · p. 57

H Chat Templates

  • The chat templates elicit profile-conditioned preference predictions for Roleplay and OpinionQA and specify how DPO training and free-response generation use persona context.Roleplay compares both A/B orders and averages probabilities after mapping them to the original orientation; OpinionQA presents options in source order. DPO uses chosen/rejected completions of the same context, while free-response generation supplies only the persona and request.Source: §H.2 · p. 58, §H.3 · pp. 59–60

I Free-Response Case Study

  • Selected free-response cases show WAPA linking stated preferences to actionable advice in ways absent from baseline responses, while retaining choices or considerations also found in the all-ground-truth-label policy.The cases cover non-digital reflection tools, sustainability and financial prudence in purchasing, and local economic support alongside cultural interests; they illustrate response differences rather than broad evaluation results.Source: I Free-Response Case Study · p. 60, Table S28 · p. 61, Table S29 · p. 62

J Further Evaluation of Free Responses

J.1 Human-Audited Subset Results

In an exploratory human evaluation, WAPA’s overall mean rank was close to Full-GT’s, though Full-GT had a higher first-place rate. The tested LLM judges did not consistently reproduce human method rankings, and DeepSeek’s rankings varied when display order was reversed.

J.1 Human-Audited Subset Results

  • WAPA’s overall mean rank was close to Full-GT’s (2.298 versus 2.309), while Full-GT had a higher first-place rate; these descriptive results do not establish a statistically significant difference.Thirteen blinded raters provided 128 rankings on an exploratory subset of 96 persona–prompt instances.Source: §J.1 · p. 64

J.2 Comparison of Human and LLM Judges

  • DeepSeek’s first-place agreement was close to the 25% uniform-random reference, while Qwen and GLM also showed limited agreement with humans on the matched subset.Human raters also disagreed substantially, so disagreement between a judge and a rater alone does not establish that the judge is wrong.Source: §J.2 · p. 64
  • Humans ranked Random last on average, while DeepSeek ranked it first; method orderings depended on the evaluator, and tested judges did not consistently reproduce human ordering.The comparison included the 96-instance human set and a matched 32-instance subset; Table S33 also reports DeepSeek results on an expanded 200-instance pool.Source: §J.2 · pp. 64–65, Table S33 · p. 65
  • DeepSeek selected different winners for 42/96 instances and returned different complete rankings for 70/96 when display order reversed, despite unchanged response content.Without same-order repeat controls, the evaluation cannot separate order effects from sampling variation and does not establish random guessing or content bias.Source: §J.2 · p. 65, Table S34 · p. 66
  • None of the tested alternative prompts improved DeepSeek’s observed pairwise agreement on its matched subset; the transferred prompt changed agreement by −3.1 percentage points (95% CI: [−11.6,+5.2]).The comparisons do not isolate rubric wording because output formats and token limits also varied, and they do not rule out improvements from other prompts.Source: §J.2 · p. 65, Table S35 · p. 66
  • DeepSeek’s 79.2% agreement with synthetic oracle preferences on fixed dataset pairs does not establish agreement with human rankings of freely generated responses.Given limited human agreement, differing method orderings, and ranking instability, the paper retains human evaluation as the primary assessment and reports LLM-judge results as exploratory.Source: §J.2 · p. 66, Table S36 · p. 66

Related papers

Loading 2610.19035v1…