Source-linked AI summary
Who Chooses How Preferences Are Aggregated? Auditing Aggregation-Rule Authority in LLM-Based Group Recommendation
Yuxuan Du
TL;DR
When reasonable aggregation rules support different joint actions, the unresolved question is who may choose how preferences are combined. The paper audits three LLMs across synthetic and empirical profiles under unspecified, user-retained, and model-delegated authority, finding that delegation enables commitment but does not determine the resulting collective outcome.
Problem
Different reasonable aggregation rules can support different joint actions, creating an interaction-level question about who may resolve the aggregation choice.
Method
The paper conducts a controlled behavioral audit of three LLMs using synthetic and empirical preference profiles under unspecified, retained, and delegated aggregation authority.
Results
Models almost never committed when users retained authority but committed in every delegated case; all three executed both witness rules correctly when directly instructed, while outcome distributions varied across models and settings.
Takeaways & Limitations
Delegation gives the model discretion over the aggregation choice, but does not determine which collective outcome follows.
Takeaways & Limitations
The experiments use single-turn tasks with two users, five options, and explicit numerical ratings, rather than larger groups, evolving preferences, or multi-turn negotiation.
Abstract
from arXiv · showhide
AI systems increasingly make joint recommendations for users with conflicting preferences. However, when reasonable aggregation rules support different actions, a further question arises: who may choose how those preferences are combined? We study this interaction-level problem as aggregation-rule authority. Using synthetic preference profiles and profiles constructed from empirical ratings, we conduct a controlled behavioral audit of three LLMs under three authority conditions: unspecified, explicitly retained by users, and delegated to the model. In cases where two witness rules supported different actions, models almost never committed when users retained authority, but committed in every delegated case. All three models executed both witness rules perfectly when directly instructed. Yet when authority was unspecified or delegated, their aggregation-consistent outcome distributions differed across models and preference settings. Together, these results separate rule-execution capability from aggregation-rule authority: delegation assigns the model discretion to resolve the aggregation choice, but does not determine which collective outcome follows.
Introduction
Joint recommendation requires translating conflicting individual preferences into one action, but different reasonable aggregation rules can support different outcomes. This paper studies who has authority to resolve that aggregation choice and how authority conditions affect LLM behavior.
- Introduction: Different aggregation rules can produce different group recommendations from the same users’ preferences.The preferences alone may not determine which tradeoff governs the final choice.
- Introduction: The paper frames authority over aggregation-rule selection as an interaction-level alignment problem.It asks how explicit authority allocation affects commitment and how outcomes vary when authority is unspecified or delegated.
- Introduction: The audit compares three LLMs across synthetic and empirical preference profiles under unspecified, retained, and delegated authority.Structural controls and diagnostics distinguish aggregation-sensitive deferral from general noncommitment and rule-execution capability.
- Introduction: Explicit retention and delegation sharply separated whether models committed on cases where witness rules supported different actions.Models almost never committed when users retained authority, but committed in every delegated case.
- Introduction: Delegating aggregation authority did not determine which collective outcome followed, because aggregation-consistent outcomes varied across models and preference settings.All three models nevertheless executed both witness rules correctly when directly instructed.
Related Work
Prior work studies aggregation rules, LLM rule execution, unconstrained recommendations, and strategy selection. This paper treats control over aggregation choice as an interaction variable linking authority, commitment, and collective outcomes.
- Related Work: Group recommender research proposes aggregation strategies that combine users’ preferences differently, with suitability depending partly on preference distributions.The paper highlights Additive Utilitarian and Least Misery as examples.
- Related Work: Prior studies examine whether LLMs can execute specified rules and whether their unconstrained recommendations resemble established aggregation strategies.Their explanations may not cleanly correspond to the observed recommendation.
- Related Work: Recent work also considers LLM selection among candidate aggregation strategies and multi-user settings with conflicting interests and authority levels.These strands place LLMs at multiple stages of preference aggregation.
- Related Work: This study makes control over aggregation choice an interaction variable, examining both model commitment and resulting collective outcomes.It distinguishes this focus from prior work on executing a prescribed rule or selecting among strategies.
- Related Work: The paper connects this problem to pluralistic alignment, where heterogeneous human preferences must be combined into collective decisions.Social-choice approaches make the combination of diverse inputs explicit.
Problem Formulation
The formulation separates rule execution, aggregation choice, and aggregation-rule authority. Core-conflict profiles expose the distinction by using two established rules whose unique winners differ for the same preference profile.
- Problem Formulation: A preference profile is the complete set of two users’ ratings for one joint-choice instance, from which the task produces one recommendation.Each user rates every option in a finite candidate set.
- Problem Formulation: Additive Utilitarian aggregation selects the option with the largest sum of individual ratings.It considers the combined ratings of both users.
- Problem Formulation: Least Misery selects the option whose lowest individual rating is largest.It gives greater weight to avoiding an outcome that leaves one user relatively dissatisfied.
- Problem Formulation: Core-conflict profiles have unique ADD and LMS winners that differ, so the ratings support different joint actions under established procedures.The two rules serve as witness rules; they need not exhaust all reasonable aggregation methods.
- Problem Formulation: Rule execution applies a specified procedure, whereas aggregation choice resolves how preferences are combined when no procedure is specified.Aggregation-rule authority identifies who may resolve that procedural choice.
Experimental Design
The experiments evaluate three LLMs on synthetic and empirical two-user rating profiles under three authority conditions, using controlled profile categories and behavioral diagnostics. Analyses measure commitment, deferral, rule-consistent outcomes, and direct rule execution while pairing comparisons by profile.
- Experimental Design: Two experimental settings each contain 1,000 profiles involving two users, five candidate options, and numerical ratings.Profiles differ in whether choosing an aggregation rule is consequential for the recommendation.
- Experimental Design: Experiment 1 uses synthetic profiles to control joint-decision structure, with 600 core-conflict profiles as the primary analysis set.Shared top and witness-rule agreement profiles serve as controls for alternative explanations of noncommitment.
- Experimental Design: Experiment 2 applies the same structures to MovieLens 32M ratings from user pairs with at least five movies rated in common.Five co-rated movies form each fixed two-user profile on a 0.5–5 star scale.
- Experimental Design: Natural leaves aggregation authority unspecified, User Retains says the model should not decide the tradeoff, and User Delegates assigns that decision to the model.User Delegates also requests a brief explanation, so its comparison with Natural reflects the complete interaction contrast.
- Experimental Design: The primary behavioral measure records whether the model commits to one unique action and, for noncommitment, whether it maps rules to outcomes or asks the user to choose the tradeoff.Observed choices are separately labeled as ADD-consistent, LMS-consistent, both, or neither.
- Experimental Design: Comparisons use the preference profile as the statistical unit, with Wilson intervals, paired risk differences, exact McNemar tests, and profile bootstraps.The design also directly instructs models on fixed profiles to test rule-execution capability and repeats authority-unspecified prompts to examine outcome stability.
Results
Explicit authority allocation sharply changed whether models committed on aggregation-sensitive profiles, while delegated authority left collective outcomes heterogeneous across models and preference settings despite perfect direct rule execution.
- Commitment Across Authority Conditions: 0.08% for GPT and 0% for Claude and Qwen under User Retains, versus 100% commitment for all three models under User Delegates.Unspecified authority yielded 93.3% for GPT, 97.9% for Claude, and 96.1% for Qwen.
- Commitment Across Authority Conditions: 99.9 percentage points for GPT and 100 percentage points for Claude and Qwen separated User Retains from User Delegates.All exact McNemar tests yielded p < 10−300, and the pattern replicated across experiments except for one GPT commitment under User Retains in Experiment 2.
- Behavior Under Retained Authority: Under User Retains, GPT mapped alternative outcomes and asked users how to combine preferences in 99.9% of responses, while Claude and Qwen did both in 100%.These behaviors accompanied near-zero commitment on core-conflict profiles.
- Behavior Under Retained Authority: Under User Retains, commitment remained high on shared-top profiles—100% for GPT, 98.5% for Claude, and 82.8% for Qwen—because both users shared one unique top option.Witness-rule agreement profiles instead produced 1.0%, 65.3%, and 20.3% commitment, respectively.
- Collective Outcomes Under Unspecified and Delegated Authority: Under delegated authority, ADD-consistent shares were 79.2%, 52.8%, and 81.0% in Experiment 1, and 58.3%, 40.5%, and 86.0% in Experiment 2 for GPT, Claude, and Qwen.All models committed on all 600 core-conflict profiles in each experiment, yet choices matching neither witness rule were rare.
- Collective Outcomes Under Unspecified and Delegated Authority: All three models selected the correct rule-defined winner on every direct diagnostic trial (1,500/1,500), although repeated identical abstract prompts changed outcomes in 8.0%, 20.4%, and 10.0% of GPT, Claude, and Qwen cases.Outcome changes were also observed under restaurant and travel presentations.
Discussion
Authority allocation changes whether models commit to a joint recommendation, but delegation does not determine which aggregation-consistent outcome follows. Outcome variation reflects model, preference, and interaction conditions rather than rule-execution inability.
- Authority Allocation as a Decision Boundary: Under User Retains, models overwhelmingly returned the aggregation choice to the user, whereas under User Delegates they closed the decision themselves.Retained authority typically produced structured deferral rather than simply suppressing assistance.
- Authority Allocation as a Decision Boundary: Natural interactions typically produced a unique recommendation without separately surfacing the unresolved aggregation choice.Evaluating only the final action can miss whether the model exercised procedural discretion.
- Structured Deferral as a Mode of Assistance: Structured deferral mapped alternative preference-combination approaches to different outcomes and asked the user which approach to use.Shared-top profiles restored commitment, showing that retained authority was not a general instruction to avoid recommendations.
- Structured Deferral as a Mode of Assistance: Witness-rule agreement did not produce a common retained-authority response across models, cautioning against treating deferral as mechanical.Agreement between ADD and LMS was insufficient to explain commitment behavior.
- Delegation Assigns Procedural Discretion, Not a Collective Outcome: Delegation settled who could resolve aggregation but not the substantive outcome: Claude and Qwen shifted toward ADD, while GPT shifted away in Experiment 1 and showed little change in Experiment 2.The direction and magnitude of change depended on the model and preference setting.
- Delegation Assigns Procedural Discretion, Not a Collective Outcome: Every model selected the correct ADD or LMS winner on every directly instructed rule-execution trial, separating execution capability from open-ended procedure choice.The repeat and presentation diagnostic additionally found that outcome distributions varied with interaction conditions.
- Limitations: The experiments cover single-turn decisions with two users, five options, explicit ratings, and constructed empirical profiles rather than larger-group or naturally occurring deliberation.Comparisons also reflect the complete tested prompts and model families, not stable behavior across updates or broader settings.
- Limitations: ADD and LMS are witness rules demonstrating divergent actions, not an exhaustive set of reasonable aggregation procedures.Their agreement establishes convergence only between those two procedures.
Conclusion
The paper examines who may resolve conflicting aggregation rules in LLM-based group recommendation through controlled audits across models and preference settings. Explicit authority allocation sharply changes commitment, while delegation leaves collective outcomes model- and setting-dependent despite perfect direct rule execution.
- The study treats authority over combining users’ preferences as a separate interaction-level question from selecting the joint action.
- Across three LLMs and two preference settings, explicit retention and delegation sharply separated whether models committed to a joint recommendation.
- Aggregation-consistent outcomes varied across models and settings despite perfect execution of both witness rules when directly instructed.