Evolution of fairness in multi-objective reinforcement learning framework

I. INTRODUCTION

• Behavioral experiments find typical offers of 40%–50% of the stake and frequent rejection of offers below 20%, contrary to self-interested ultimatum-game predictions.In the conventional model, proposers offer the minimum and responders accept any positive offer.Source: §I · p. 1
• Prior fairness studies examine population structure, roles, noise, reputation, spite, and empathy, with most using imitation learning based on neighbors’ higher-payoff actions.The paper characterizes imitation learning as outward-looking because it relies on social comparison and behavioral imitation.Source: §I · p. 1
• Single-objective reinforcement learning maximizes accumulated rewards, but the authors argue it does not capture decisions involving competing objectives or moral concerns.They ask whether a multi-objective framework can offer insights into human behavior.Source: §I · p. 2
• The proposed multi-objective Q-learning ultimatum game combines personal payoff and fairness objectives, yielding higher fairness than a single-objective baseline and non-monotonic responder strategy changes.The authors examine evolutionary dynamics and Q-tables to investigate the strategy reversal’s mechanism.Source: §I · p. 2
• The paper proceeds from the spatial ultimatum-game model to evolutionary results, mechanism analysis, an asymmetric extension, and conclusions.Source: §I · p. 2

II. MODEL

• The study models a two-player ultimatum game with fixed proposer and responder roles and a total endowment normalized to 1; the proposer chooses an offer and the responder sets a minimum acceptable offer.Source: §II · p. 2
• Each player’s state reflects both players’ previous-round actions, yielding nine states and three available actions in each Q-table.The same state-action structure is used for the payoff and fairness Q-tables.Source: Table I · p. 2, §II · p. 3
• Players track individual payoff and distributive fairness as separate objectives, with a reward r when their payoffs are equal and zero otherwise.The payoff objective reflects economic incentives, while fairness is linked to emotional or moral incentives.Source: §II · p. 3
• A weighted sum of payoff and fairness Q-values guides action selection; ω sets fairness preference, with ω = 0 reducing to the single-objective case.Higher ω increases reliance on the fairness Q-table.Source: §II · p. 3
• After each round, players update both objective-specific Q-tables using a Bellman rule, while the aggregate action-selection value is recomputed from them.The simulations use ε = 0.01, α = 0.1, and γ = 0.9.Source: §II · p. 3

• In the cited single-objective benchmark, fair strategies dominate for both roles, with adoption probabilities of approximately 85%.The remaining probability is shared with the rational strategies, according to Ref..Source: §III · p. 4
• As fairness weight and reward increase, proposers increasingly favor the fair offer, while responders shift from the intermediate to low and then back to intermediate acceptance threshold.At ω →1, all three responder thresholds occur with comparable probabilities; overgenerous offers and expectations remain rare except for responders at this extreme.Source: §III · p. 4
• Increasing ω raises proposer adoption of the fair offer from about 85% toward near certainty, while responder adoption shifts qm upward, then ql surges, and qm rises again.Fig. 2 plots action-adoption probabilities against ω for r = 0.1, 0.3, and 0.5, separately for proposers and responders.Source: §III · p. 4, Figure 2 · p. 5
• The adoption curves are qualitatively similar across r = 0.1, 0.3, and 0.5, but larger r shifts the transition onset to lower ω.Source: §III · p. 5, Figure 2 · p. 5
• Increasing r also promotes the fair proposer offer and produces the responder reversal, though the proposer’s rise is absent at ω = 0.2 and the reversal narrows at high ω.The paper therefore focuses its subsequent analysis on ω, since r has a qualitatively similar impact.Source: §III · p. 5

Evolution of action, state, and reward

• High actions ph and qh decline after the transient because they often cause failed divisions, yielding low payoffs and poor fairness outcomes.Figure 4 tracks action fractions, state fractions, and player payoffs across the three pressure levels.Source: §IV · p. 5, Figure 4 · p. 6
• At weak pressure (ω = 0.2), pm and qm gradually dominate but pl and ql persist, leaving unfair (l, l) splits and average fairness reward below r = 0.2.The remaining low-low states lower responder reward relative to proposer reward.Source: §IV · p. 5
• At intermediate pressure (ω = 0.5), responders more often choose ql while proposers exclusively choose pm, producing full fairness with equal rewards.The abundant (m, l) state reflects responders accepting occasional low offers rather than proposers lowering their fair offer.Source: §IV · pp. 5, 7
• At strong pressure (ω = 0.8), ql and qm reverse their intermediate-pressure densities, and the outcome is fully fair with equal player rewards.The resulting states are mostly (m, m), with a few (m, l).Source: §IV · p. 7

Analysis of Q-tables

• Proposers generally prefer pm over pl in aggregate Q-tables, while fairness-oriented values can outweigh payoff-table preferences for pl in some states.At s4, proposers favor pl when ω < 0.3 but switch toward pm as pressure rises.Source: §IV · p. 7
• Responder preference changes explain the reversal: qm is typical at weak pressure, ql is preferred in dominant states at intermediate pressure, and no clear preference remains at strong pressure.The preference in s9 is less developed because that state is rarely visited.Source: §IV · p. 7

V. EXTENSION

• The asymmetric design gives proposers payoff-only learning and a last-round proposer-action state, while responders retain two-objective learning.Responders are modeled as more sensitive to fairness, and they directly observe the offer while proposers may not know the acceptance threshold.Source: §V · p. 8
• As responder fairness pressure rises, low-action adoption gives way to medium actions, while high actions remain near zero across the parameter domain.The proposer uses payoff-only learning; the responder retains two-objective Q-learning.Source: §V · p. 8
• In the single-objective case, this asymmetric state setup completely suppresses fairness, and accurate state specification requires further experimental evidence.The comparison is with Fig. 1 and Ref.; the authors relate the result to prior studies.Source: §V · p. 8

VI. CONCLUSION

• Separate Q-tables for payoff and fairness objectives are integrated through a tunable fairness-pressure coefficient into an aggregated Q-table guiding decisions.The asymmetric extension shown in Fig. 7 uses payoff-only learning for proposers and two-objective learning for responders.Source: §VI · p. 8, Figure 7 · p. 9
• The two-objective learning framework promotes higher fairness than a single-objective baseline, with full fairness emerging above a moderate fairness-pressure threshold.Responders lower their acceptance threshold from low to moderate pressure, then raise it as pressure increases further.Source: §VI · p. 9
• The authors caution that reinforcement learning remains too immature to serve as a comprehensive theoretical model of real-world human behavior.This frames the approach as a useful lens rather than a comprehensive account.Source: §VI · p. 9
• The authors propose MORL as a framework for studying cooperation, trust, and other social phenomena beyond fairness.They describe it as an extensible generalization of single-objective reinforcement learning.Source: §VI · p. 9