Mechanism Design for LLM Fine-tuning with Multiple Reward Models

1 Introduction

• Separate models for every group are often impractical because of computational costs and privacy concerns, motivating a unified model that aggregates distinct preferences.Multiple-Objective RLHF aims to integrate multiple preferences into one model.Source: §1 · p. 1
• Profitable preference misreporting can steer the fine-tuned model toward a group’s preferred outcome while distorting the objective for the overall community.The introduction illustrates that a group can gain utility by reporting a polarized preference instead of its true preference.Source: §1 · p. 2, Figure 1 · p. 3
• The paper models a fine-tuning provider and preference-reporting groups as a mechanism design problem, focusing on social-welfare-maximizing training rules with regularization.The mechanism includes a training rule and a payment rule for groups with different preferences and sizes.Source: §1 · p. 2
• SW-Max rules alone can permit profitable misreporting; the paper extends VCG payments, characterizes implementation conditions, and shows approximate DSIC under perturbed inputs.The paper also reports experiments validating profitable misreporting and the efficacy of its mechanism in incentivizing truthful reporting.Source: §1 · p. 2
• The analysis complements prior RLHF incentive studies by examining representative training rules and payment equivalence, while distinguishing preference misreporting from bid monotonicity.The cited preference aggregation mechanism of Duetting et al. [25] addresses bid monotonicity but not strategic preference misreporting.Source: §1 · p. 2

2 Preliminaries

• An LLM maps each prompt to a probability distribution over output sequences of length at most K, and its parameters are assumed able to express any function in the model space.The theoretical model considers a fixed prompt independently and suppresses prompt notation.Source: §2 · p. 3
• The model’s theoretical analysis focuses on a fixed prompt scenario, with the LLM output represented as a distribution over sequences up to length K.The prompt notation is omitted for simplicity.Source: §2 · p. 3
• A reward model maps a prompt-response pair to a real-valued satisfaction score, and the analysis assumes non-negative rewards under either sum-to-one or maximum-one normalization.The paper states that its results hold under both normalization schemes unless otherwise specified.Source: §2 · p. 3

3 Formulation of the RLHF Game

• Each group’s private information consists of its reward model and integer group size, with group size included to account for scale differences and possible size over-reporting.The group-size domain has a publicly known upper bound, and the model includes the single-agent case.Source: §3 · p. 4
• A group values a model by its group size multiplied by the expected reward its reward model assigns to generated sequences.The valuation is the expected reward under the LLM’s output distribution.Source: §3 · p. 4
• The provider’s training rule maps reported reward models, reported group sizes, and an initial model to a fine-tuned model, while the payment rule outputs group charges.The provider can choose zero payments, and the training rule selects a model maximizing its stated objective.Source: §3 · p. 4
• Groups report their information, receive access to the resulting fine-tuned model, and pay once; their utility is valuation minus payment.Reports need not equal true preferences or sizes, so misreporting can distort the training outcome.Source: §3 · p. 4
• DSIC requires truthful reporting to maximize each group’s utility regardless of other reports, while IR requires truthful reporting to yield non-negative utility.A payment rule implements a training rule when the resulting mechanism satisfies the stated property.Source: §3 · p. 4

4.1 Necessity of Payment Rule

• When a group has at least two distinct positive reward values, it can profitably shift weight toward preferred outcomes; truthful reporting is also often suboptimal under the stated smoothness condition.For the zero-minimum case, Theorem 4.3 gives a sufficient condition: a nonzero function t implies truth-telling is not optimal.Source: §4.1 · pp. 5–6
• For strongly convex, C2-smooth regularizers and a zero minimum reward, a nonzero t likewise makes truthful reporting suboptimal.The analysis assumes generated outcomes have positive likelihood and excludes cases where outcomes barely change with input.Source: §4.1 · pp. 5–6

4.2 Affine Maximizer Payment

• The affine maximizer payment extends VCG to additive SW-Max objectives and makes truthful reward-model and group-size reports dominant, with individual rationality and nonnegative payments.The ordinary VCG payment cannot be applied directly because SW-Max objectives include a regularization term.Source: §4.2 · p. 6
• Every continuous SW-Max training rule satisfies payment equivalence; continuity holds for the stated KL- and χ2-divergence regularizers.With public group sizes, all implementable rules satisfy payment equivalence; the general result uses a report that connects different group sizes.Source: §4.2 · p. 7
• For continuous SW-Max rules, nonnegative or individually rational DSIC payments face necessary constraints, and the affine maximizer payment serves as a lower bound that other rules must approach on some inputs.The lower-bound implication concerns payment rules satisfying the specified properties, not every payment rule.Source: §4.2 · p. 7

4.3 Approximate Valuation

• With noisy reward-model inputs within ε of reported values, the affine maximizer mechanism limits each group’s misreporting gain to at most 2wiε.The authors note that truth-telling may be favored when finding and changing a profitable report costs more than that bound.Source: §4.3 · p. 8

4.4 Efficient Implementation of the Mechanism

• Exact payment computation requires n additional trainings, so two heuristics restrict the maximization to a smaller candidate set: intermediate models or early-stopped training.Intermediate models preserve nonnegative payments but do not guarantee DSIC; early-stopped training preserves DSIC but may yield negative payments.Source: §4.4 · pp. 8–9
• The heuristics trade computational efficiency against theoretical guarantees, and the paper leaves rigorous efficiency-loss analysis and an impossibility result for future work.Payments satisfying DSIC, individual rationality, and nonnegativity must closely approximate the affine maximizer payment on some inputs.Source: §4.4 · p. 9

Models and Datasets

• The experiments use Helpful Assistants and Reddit Summary with Llama-2 7b, modeling harmlessness versus humor and faithfulness versus summary quality.Each game has two groups with synthetic size pairs (3, 7), (5, 5), or (7, 3).Source: §5 · pp. 9–10

Implementation Details

• The implementation uses KL-divergence and a finite candidate set of reward-model hybrids formed with Rewarded Soups to reduce training costs.The candidate models are produced by training on individual reward models and then combining them.Source: §5 · p. 10

Misreporting Strategies

• The study examines naïve group-size exaggeration and strategic preference mixing, which respectively require no knowledge of other groups and some knowledge of preference conflicts.Both strategies increase a group’s influence in the training process when α or β is increased.Source: §5 · p. 10

Result Analysis

• Increasing α or β raises group valuation, but the affine maximizer payment offsets those gains, making truthful reporting yield the highest utility in all tested cases.The strategies tested are group-size overstatement and shifting a reported reward model toward the other group’s preferences.Source: §5 · p. 10

6 Conclusion and Future Work

• The paper derives sufficient conditions for payment equivalence and necessary conditions for implementing SW-Max rules, and reports approximate DSIC under perturbed input.Experiments in real-world LLM setups are also reported to show effective incentives for truthful reporting.Source: §6 · p. 10
• Future work could study more general training rules, simpler implementability conditions, and payment equivalence for those rules.The paper notes cycle monotonicity is necessary and sufficient for implementability, but complex to validate.Source: §6 · p. 10
• Future work could examine preference aggregation across multiple models, particularly with diversity considerations.Source: §6 · p. 10
• Further directions include balancing computational efficiency with incentive compatibility and applying mechanism design to API pricing, RAG, and prompt engineering.Source: §6 · pp. 10–11

Limitation

• The paper mainly studies SW-Max training rules and their theoretical properties; further work could extend the model to DPO, where groups provide data pairs rather than reward models.Source: Limitation · p. 16

A.1 RLHF with Multiple Reward Models

• Prior multiple-reward-model studies develop preference-combination methods, expand Pareto frontiers, balance objectives, or ensemble models for robustness and reward-hacking mitigation.Source: §A.1 · p. 16
• Unlike work that often assumes access to humans’ actual preferences, this paper considers incentives for rational agents to report preferences truthfully.Source: §A.1 · p. 16

A.2 Multi-parameter Auctions

• The paper identifies payment equivalence and revenue-maximizing solutions for SW-Max training rules in LLM fine-tuning as contributions not addressed by the surveyed auction work.Source: §A.2 · p. 16

A.3 Game Theory and LLMs

• Other game-theory and LLM research studies bargaining in fine-tuning, auctions for content placement in summaries, social choice in alignment, and online advertising.Source: §A.3 · p. 16
• Related work also uses game-theoretic concepts to train LLMs and evaluates models in rationality, matrix games, and strategic games including auctions and Diplomacy.Source: §A.3 · p. 16

B Omitted Proofs in Section 4.1

• If si = 1, truthful reporting is optimal regardless of other groups’ reports; the proof explains that the group gets equal utility from all training outcomes.Source: B Omitted Proofs in Section 4.1 · p. 17
• When si ≥ 2 and rmi > 0, a group has a strategy yielding strictly higher utility than truthful reporting regardless of other groups’ reports.The theorem’s stated setting assumes an α-strongly convex, C2-smooth divergence function.Source: B Omitted Proofs in Section 4.1 · pp. 17, 19
• For group i, when si ≥ 2, the aggregate utility change, computed as the sum over x of (LLMθ′(x) − LLMθ(x)) rmi(x), is strictly positive.Source: B Omitted Proofs in Section 4.1 · p. 19
• For the maximum normalization case, if there exists x1 ∈ T* such that t(x1) ≠ 0 and 0 < rmi(x1) < 1, truthful reporting is not the optimal strategy.Source: B Omitted Proofs in Section 4.1 · p. 20
• Under summation normalization, if there exists x1 ∈ T* with t(x1) < 0 and 0 < rmi(x1) < 1, truthful reporting is not the optimal strategy.Source: B Omitted Proofs in Section 4.1 · p. 20

• For every ψ ∈ ΨSW, the affine-maximizer mechanism satisfies DSIC and individual rationality, with non-negative payments.The proof derives truthful utility as non-negative.Source: C Omitted Proofs in Section 4.2 · pp. 23–24
• When agent numbers are publicly fixed at 1 and agents report only reward models, all implementable training rules satisfy payment equivalence.The assumed non-negative, normalized reward-model domains are connected under both summation and maximum normalization.Source: C Omitted Proofs in Section 4.2 · p. 24
• SW-Max training rules using KL-divergence or χ²-divergence regularization are continuous.For χ²-divergence, the proof assumes λ is large enough to ensure positive model probabilities on T∗.Source: C Omitted Proofs in Section 4.2 · pp. 24–25
• A continuous training rule has payment equivalence if a reward model leaves its training outcome unchanged across agent weights; under maximum normalization, that model must be 1.The proof establishes the condition using the rule’s continuity and a path argument.Source: C Omitted Proofs in Section 4.2 · pp. 26, 29
• For continuous SW-Max rules, any DSIC payment rule that is always non-negative must charge at least the affine-maximizer payment.Theorem 4.11 states this lower bound for all inputs.Source: C Omitted Proofs in Section 4.2 · p. 30

C Omitted Proofs in Section 4.2

• In an extreme two-output case, group i’s utility is at most ε, so IR requires g_i ≤ ε, equivalently p_i ≤ p_i^AFF.The group prefers x1, while all other groups prefer x2; the initial model assigns probability ε to x1. The optimality condition bounds the final probability of x1 by its initial value, and p_i^AFF is non-negative.Source: C Omitted Proofs in Section 4.2 · p. 30

D Omitted Proofs in Section 4.3

• When each noisy reward model differs from its true model by at most ε, the training outcome’s distance from the optimum is bounded.The supplied statement gives the bounded-distance conclusion but not its bound’s formula.Source: D Omitted Proofs in Section 4.3 · pp. 30–31
• The proof compares outcomes optimized for noisy and true reward-model inputs, using bounded reward differences and each outcome’s optimality.The appendix identifies these ingredients in its derivation of the outcome-distance bound.Source: D Omitted Proofs in Section 4.3 · pp. 31–32
• Each group can gain at most 2w_iε by misreporting its reward model under p_AFF with a SW-Max training rule.The proof uses a bound on reward-model distance and the optimality of the training outcome for each input.Source: D Omitted Proofs in Section 4.3 · pp. 30–32

E Further Discussion on General Training Rules

• The paper focuses on mainstream rules that maximize social welfare under regularization, rather than rules such as Nash Social Welfare maximization.The latter are described as focusing more on fairness issues.Source: E Further Discussion on General… · pp. 32–33
• For general training rules, implementability holds if and only if the rule satisfies cycle monotonicity.The paper identifies cycle monotonicity as a generalization of single-parameter monotonicity.Source: E Further Discussion on General… · p. 33
• Validating cycle monotonicity is complex, motivating further work on more concise implementability conditions for general rules or subsets of them.The discussion also places general training-rule properties outside this paper’s scope.Source: E Further Discussion on General… · pp. 33–34

Synthetic RLHF Game

• The synthetic RLHF game samples five groups’ normalized preferences over ten outcomes and assigns group sizes from 1 to 10.The preferences are sampled from U[0, 1]^10, and group sizes from {1, 2, …, 10}^10.Source: F Additional Experimental Results · p. 34

Misreporting strategy

• A misreporting strategy raises valuation but decreases utility under the payment mechanism, while no strategy improves utility over truthful reporting.The experiment holds other groups’ reports fixed and varies the deviation extent ϵ.Source: Table 1 · p. 34, F Additional Experimental Results · p. 34

More Complex Preferences

• The complex-preference experiments use both simulated three-group preferences and a scenario with linear combinations of two reward models.The first scenario draws on data reported in [70]; the second assigns groups different Helpfulness–Harmlessness mixtures or Humor.Source: F Additional Experimental Results · p. 35
• Across the reported complex-preference scenarios, truthful reporting is among the optimal strategies under the mechanism.The scenarios include groups with distinct Helpfulness, Harmlessness, and Humor preferences, as well as preferences combining Helpfulness and Harmlessness.Source: F Additional Experimental Results · p. 35