Source-linked AI summary
What preferences can - and cannot - predict in multi-agent online learning
Omar Abbadi, Rida Laraki, Panayotis Mertikopoulos
TL;DR
The paper asks how far preference graphs can predict stable long-run outcomes in multi-agent learning. It analyzes regularized dynamics, identifies when preferences suffice or fail, and introduces a payoff-based condition guaranteeing stability for arbitrary strategy spans.
Problem
The paper asks which set-valued long-run outcomes can be stable and whether preference structure alone determines them beyond subgames.
Method
The paper relates preference-graph stability to regularized learning dynamics, constructs a separating three-player example, and studies a payoff-based resilience condition.
Results
Preference graphs constrain stable outcomes and characterize asymptotic stability for subgames, but preferential stability can coexist with dynamic instability beyond subgames.
Takeaways & Limitations
Preference information provides a combinatorial blueprint for regularized learning outcomes, while payoff-based resilience can guarantee asymptotic stability for arbitrary strategy spans.
Takeaways & Limitations
The proposed resilience condition is sufficient but not necessary, leaving open a simple payoff-based characterization of all asymptotically stable sets.
Abstract
from arXiv · showhide
We examine the interplay between ordinal, preference-based solution concepts in games and the long-run behavior of game dynamics, asking in particular to what extent the combinatorial data of a game -- its preference graph -- determine the outcomes of no-regret learning dynamics -- such as follow-the-regularized-leader (FTRL). In one direction, we show that the skeleton of every dynamically stable set (i.e. the set of pure profiles it contains) must also be preferentially stable, that is, it must be closed under profitable deviations. We then ask the converse question: when do preferences determine the long-run behavior of the players' learning dynamics? We begin by showing that preferences characterize asymptotic stability in the case of subgames -- i.e. subsets of pure profiles obtained by restricting players' action sets. Beyond this case however, the equivalence between dynamic and preferential stability collapses: concretely, we construct a three-player game with a preferentially stable set whose span is dynamically unstable, showing in this way that preferences do not suffice as a criterion of dynamic stability. We then bridge this gap via the notion of resilience under aggregate deviations, an easy-to-check payoff-based condition that guarantees asymptotic stability of arbitrary spans of pure strategies.
1. Introduction
The paper studies when ordinal preferences, encoded by a preference graph, determine stable long-run outcomes under FTRL and when cardinal payoffs matter. It establishes necessary preference-based restrictions, exact characterization for subgames, a counterexample beyond subgames, and a payoff-based sufficient condition for stability.
- Motivation: FTRL models no-regret learning in which players respond to cumulative rather than instantaneous payoffs, making asymptotic behavior difficult to characterize in general games.The paper focuses on FTRL as a widely used class of no-regret procedures for learning in games.
- Preference-based restrictions: Any asymptotically stable set has a skeleton closed under better replies, and every attractor contains the mixed region spanned by each connected set of pure profiles it contains.If the preference graph is strongly connected, FTRL admits no proper attractor.
- Subgames: For subgames, asymptotic stability is characterized exactly by closedness under better replies.This also gives a sharp characterization of attractors in weakly acyclic games.
- Limits of preferences: Beyond subgames, a set of pure profiles can be closed under better replies while its span remains unstable under regularized learning, even from full-support initialization.Thus preferential and dynamical stability need not coincide.
- Payoff-based stability: The paper introduces resilience to aggregate deviations (rad), a payoff-based condition guaranteeing asymptotic stability for arbitrary spans of pure strategies.Radness generalizes pure Nash equilibrium setwise and refines closedness under better replies using cardinal payoff information.
2. Preliminaries
The paper models finite normal-form games through pure and mixed strategy profiles, with payoffs extended multilinearly to mixed strategies. It also defines the payoff field and standard equilibrium concepts, including Nash and strict Nash equilibria.
- Game and strategies: A finite normal-form game consists of players with finite action sets and payoff functions over pure action profiles.The pure-profile space is A := ∏_i∈N A_i.
- Game and strategies: Each player’s mixed strategy is a probability distribution over actions, and mixed profiles form the product strategy space X.Pure actions and profiles are identified with the corresponding simplex vertices.
- Payoff field and equilibria: Payoffs extend multilinearly to mixed profiles, with the payoff field recording each action’s expected payoff against opponents’ mixed strategies.For player i and action α_i, v_iα(x) = u_i(α_i, x_−i), and ⟨v_i(x), x_i⟩ = u_i(x).
- Payoff field and equilibria: A Nash equilibrium is a mixed profile where no player can gain from unilateral deviation, while a strict Nash equilibrium is a pure profile where every player strictly prefers the equilibrium action.The Nash condition compares equilibrium payoff with every unilateral alternative strategy.
3. Preferences, learning and stability
This section formalizes preference-based stability through preference graphs, clubs, sink equilibria, spans, and skeletons, then introduces FTRL and its induced strategy dynamics. It also defines dynamic stability and attraction for learning trajectories and strategy flows.
- Preferences and ordinal stability: The preference graph records weakly profitable unilateral deviations between comparable pure profiles, encoding ordinal incentives as directed arcs.Outgoing edges from a profile identify comparable profiles preferred by the deviating player.
- Preferences and ordinal stability: A club is closed under weakly profitable deviations, while an s-club is closed under strictly profitable deviations except when the deviation is tied.Proper club and s-club sets are required to be nonempty proper subsets of the pure-profile set.
- Preferences and ordinal stability: A nonempty club set that is strongly connected is a sink equilibrium and therefore a minimal club set.Strong connectedness requires directed unilateral-improvement paths between every pair of vertices to remain inside the set.
- Regret and FTRL: FTRL selects mixed actions by trading off cumulative-payoff scores against a regularization term that smooths dynamics and encourages exploration.The regularizer also determines the geometry of the induced learning dynamics through the choice map.
- Strategy dynamics: With steep regularizers, supports remain fixed and trajectories stay in the relative interior of their initial face, yielding globally well-posed, face-invariant strategy dynamics under mild regularity assumptions.The facewise vector fields coincide on adjacent-face intersections and glue into a globally Lipschitz vector field.
- Dynamic stability and attraction: Asymptotic stability combines Lyapunov stability with attraction, while strategy-flow attractors are invariant asymptotically stable sets.The strategy-flow definition permits nearby initializations on all of X, whereas FTRL asymptotic stability restricts initializations to the choice-map image.
4. Implications of dynamic stability
The section links preference-graph properties to the long-run stability of strategy-space regions under regularized learning. It shows that dynamic attraction or FTRL stability imposes closure conditions on pure profiles, while strong preference connectivity constrains attractor shape.
- The section’s guiding framework distinguishes ordinal stability properties of preference graphs from dynamical stability properties of strategy-space regions.The preference graph imposes constraints on which regions can be long-run stable under regularized learning, while the proofs are deferred to Appendix F.
- Attraction under the strategy dynamics requires the attractor’s skeleton to be club.This establishes that attracting regions cannot contain outgoing better replies among their pure profiles.
- Stability under FTRL likewise requires the skeleton of a stable set to be s-club.A strict better reply would eventually shift the deviating player’s mixed action outside any sufficiently small neighborhood, contradicting stability.
- Strong connectivity in the preference graph forces any strategy-dynamics attractor containing a strongly connected set H to include span(H).Consequently, when the preference graph is strongly connected, the strategy flow has no proper attractor.
- The attractor-shape result follows from preference connectedness percolating to mixed regions through chain transitivity.The proof proceeds inductively from subfaces to whole faces, using that no asymptotically stable set lies entirely in a strategy-flow face’s interior.
5. Implications of preferential stability
For subgames, preferential stability exactly characterizes asymptotic or attractor stability under the relevant dynamics. Beyond product-structured subgames, preferentially stable sets can have dynamically unstable spans, so preference data alone do not suffice.
- Subgames: For a club subgame B, span(B) is asymptotically stable under FTRL.The proof uses a Fenchel-gap energy that vanishes exactly on span(B) and decreases proportionally to probability mass outside B.
- Subgames: With no ties, a subgame B is club if and only if span(B) is asymptotically stable under FTRL.Without the no-ties assumption, clubness is still equivalent to being an attractor of the strategy dynamics.
- Weak acyclicity: In weakly acyclic games without ties, the minimal attractors of strategy dynamics are exactly the strict Nash equilibria.Weak acyclicity requires every action profile to have a finite better-reply path ending at a pure Nash equilibrium.
- Beyond subgames: The counterexample shows that preferential stability of a non-subgame H may fail to control the dynamic stability of span(H).The separation relies on the failure of the product structure that subgames provide.
- Beyond subgames: A 2 × 2 × 2 game has a unique proper club set H whose span is not stable under FTRL.A profitable deviation toward an excluded face can occur before other players move within the span, and the minimal attractor containing H is not itself a span of pure profiles.
6. Recovering dynamic stability
Preferences alone may not determine dynamic stability, so the paper introduces payoff-dependent resilience to aggregate deviations. Strict resilience guarantees attractivity of the corresponding strategy span, while a rad-club condition yields the analogous result for replicator dynamics.
- Motivation and definitions: Payoff-dependent resilience to aggregate deviations restores stability when preferential information is too coarse to determine dynamic behavior.The condition uses payoff magnitudes, interpreted as preference-graph edge weights, rather than ordinal preferences alone.
- Motivation and definitions: If H is rad, then H is s-club; if H is s-rad, then H is club.Thus, aggregate-deviation resilience implies the corresponding preferential closure property.
- Computation: Radness can be checked in O(|N| |H| |A \ H|) steps, while finding a rad set takes O(|N| |A|2) time.The algorithm constructs a payoff-flux graph and identifies rad sets as those with no outgoing arcs.
- Strategy dynamics: If H is s-rad, span(H) is an attractor of the strategy dynamics.The result is established using an energy function that vanishes exactly on the span and dissipates along FTRL orbits.
- Replicator dynamics: If H is a rad club subset, span(H) is an attractor of the replicator dynamics.In games without ties, Proposition 3 therefore implies that the span of every rad set is asymptotically stable under replicator dynamics.
7. Concluding remarks … B.1. Sink equilibria.
The paper concludes that preference structure constrains but does not fully determine stable outcomes of regularized learning. It situates these results within attractor theory, preference-graph analysis, and sink-equilibrium dynamics while highlighting unresolved questions about stability and limiting behavior.
- 7. Concluding remarks: Preference structure offers a combinatorial blueprint constraining stable outcomes of regularized learning, but it is insufficient to determine them completely.The authors also note that radness is sufficient but not generally necessary for asymptotic stability.
- 7. Concluding remarks: A full account of players’ limiting behavior remains elusive, even among games sharing the same preference graph and lacking proper attractors.Figure 5 presents four such games with markedly different limiting behaviors.
- 7. Concluding remarks: The authors conjecture that asymptotically stable limit sets of replicator dynamics or FTRL may lie on the boundary of the game’s strategy space.They state that establishing this likely requires new ideas and remains future work.
- Appendix A. Further related work: Attractor analysis originates in evolutionary game theory, where replicator dynamics model evolutionary selection and long-run learning behavior is understood as set-valued.Ritzberger and Weibull studied asymptotic stability of faces under evolutionary dynamics.
- Appendix A. Further related work: Replicator dynamics arise as the continuous-time limit of exponential weights, while regularized dynamics such as FTRL unify no-regret learning.This connects identifying replicator attractors to identifying stable long-run outcomes more broadly.
- Appendix A. Further related work: In special game classes, regularized learning may cycle rather than converge, while zero-sum replicator dynamics have the span of the sink equilibrium as their unique global attractor.Harmonic games are described as Poincaré recurrent in the interior under FTRL and therefore lacking proper attractors.
- Appendix A. Further related work: Preference-graph research links minimal replicator attractors to sink equilibria, with established results that minimal attractors exist and always contain sink equilibria.Biggar and Shames also proved a weaker form of Lemma F.2 for chain transitive sets containing a strongly connected set.
- B.1. Sink equilibria.: Sink equilibria are sink strongly connected components of the preference graph: nonempty sets closed under better replies and strongly connected, equivalently minimal club sets.Once a better-reply path enters one, it can continue inside but cannot leave; probabilistically, preference-edge random walks provide a complementary interpretation through recurrent classes.
B.2. Examples. … Appendix D. Regularizers and choice maps
The paper illustrates preference-graph concepts through games with distinct equilibrium and cycling structures, then records dynamical-systems definitions and regularizer assumptions used in the analysis.
- B.2. Examples.: The Prisoner’s Dilemma has an acyclic preference graph whose paths all lead to (D, D), the unique strict Nash equilibrium.The sink vertex is therefore (D, D).
- B.2. Examples.: Matching Pennies has a strongly connected directed 4-cycle, making the whole graph its unique club set.The game is zero-sum.
- B.2. Examples.: Jordan’s Matching Pennies exhibits persistent adaptive-play cycling on a red 6-cycle, while its sink equilibrium is rad but not s-rad.For the sink H, Φ(β, α) ∈ {0, −2} for all α ∉ H and β ∈ H.
- B.2. Examples.: Shapley’s 3 × 3 game has a sink equilibrium whose span is a directed cycle and is rad but not s-rad.The calculation gives Φ(β, α) ≤ 0, with equality for some α and β.
- B.2. Examples.: A club subgame can fail to be rad: for β = (A, A) and α = (C, B), Φ(β, α) = 2 > 0, although club subgames are asymptotically stable.Thus radness is not necessary for asymptotic stability.
- Appendix C. Attractors and chain transitivity: The appendices define stability, attraction, and asymptotic stability for flows, with asymptotic stability requiring both stability and attraction.An attractor is a nonempty compact invariant asymptotically stable set.
- C.2. Chain transitivity.: Internal chain transitivity is equivalent to invariance together with the absence of any proper attractor in the restricted flow.Pseudo-orbits define the reachability relation underlying chain transitivity.
- Appendix D. Regularizers and choice maps: Appendix D assumes decomposable, continuous, interior-smooth regularizers with strong convexity, steepness conditions, and a convex conjugate used for choice maps.The regularizers are defined on the simplex through component functions θ_i.
D.1. Choice maps. … Appendix E. Strategy dynamics
The appendices characterize regularized choice maps, Fenchel couplings, and standard regularizers, then establish well-posed continuous-time FTRL dynamics and a facewise strategy-space description. Key properties include Lipschitz continuity, strong-convexity-based primal-dual distance, and global uniqueness of score-space trajectories.
- D.1. Choice maps.: Strong convexity makes each regularized choice map Q_i single-valued, while Proposition D.1 establishes Lipschitz continuity and identifies Im Q_i with dom ∂h_i.The map is also invariant under adding a constant multiple of the all-ones vector to scores.
- D.1. Choice maps.: If h_i is steep, the choice map’s image is the relative interior, and its restriction to Y_i/span{1} is injective.Equivalently, two scores generate the same choice only when they differ by c1.
- D.2. Bregman divergence and Fenchel coupling.: Under differentiability, the generalized directional derivative defining the extended Bregman divergence reduces to the usual expression ⟨∇h_i(x_i), p_i−x_i⟩.This connects the extended construction to standard Bregman divergence on differentiable points.
- D.3. Examples of regularizers.: The entropy regularizer has K_i=1 and s_i(p)=p, is steep, and induces the softmax choice map and Kullback–Leibler divergence.These properties are presented as a standard example satisfying the appendix assumptions.
- D.3. Examples of regularizers.: The power family includes both steep and non-steep cases, with K_i=1 and s_i(p)=p^(2−λ); λ=2 gives Euclidean regularization and λ=1/2 gives square-root regularization.Entropy is recovered in the limit λ→1 up to affine terms on the simplex.
- Appendix E. Strategy dynamics: For continuous-time FTRL, v∘Q is Lipschitz continuous, so ẏ=v(Q(y)) has a unique global solution y(·):R→Y from every initial condition.The proof uses multilinearity and boundedness of v, Lipschitz continuity of Q, and the Cauchy–Lipschitz/Picard–Lindelöf theorem.
- Appendix E. Strategy dynamics: On intervals with constant support B_i, the strategy dynamics follow a facewise ODE derived from KKT conditions and differentiated score dynamics; in the steep case, this applies globally with B_i=A_i.The appendix then seeks a globally defined vector field on X, requiring additional regularity of s_i.
E.1. Construction of the strategy flow.
Under Assumption E.1, the strategy field generates a unique globally defined, continuous flow that preserves every face of the strategy space. This flow is a globally Lipschitz patchwork of facewise FTRL dynamics, and its time reversal corresponds to the strategy flow of the negated game.
- Regularity assumptions: Assumption E.1 requires each regularizer derivative s_i to extend globally Lipschitzly on [0,1] with s_i(0)=0, and it implies that h_i is steep.All steep regularizers in Appendix D.3 satisfy this assumption.
- Regularity assumptions: Under Assumption E.1, each choice map π_i is Lipschitz continuous because the normalizing denominator S_i is uniformly bounded below by some σ_i>0.The lower bound follows from continuity and strict positivity of s_i on (0,1].
- The strategy flow: The strategy ODE admits a unique global solution for all t∈R, defining a continuous flow (Θ_t) that is invariant on every face of X.Global Lipschitzness of the strategy field yields existence and uniqueness, while zero coordinates remain zero.
- The strategy flow: The reversed flow Θ−t is generated by −F and coincides with the strategy flow of the game with negated payoffs −u_i.Replacing the payoff field v by −v in the strategy-field construction produces exactly −F.
- Facewise FTRL dynamics: On each face F=span(B), strategy-flow trajectories coincide with the corresponding facewise FTRL-B orbits, making the global flow a globally Lipschitz patchwork of facewise dynamics.The face-restricted choice map Q_B identifies the relevant FTRL dynamics.
Appendix F. Omitted proofs from Section 4 · F.1. Stability and attraction.
The appendix establishes core FTRL properties used in Section 4: dominated mixed strategies become extinct, and asymptotically stable sets must intersect pure-profile vertices. It then proves that stable sets contain pure profiles closed under better replies, using face-invariance and dominated-strategy elimination.
- Appendix F. Omitted proofs from Section 4: Under FTRL, any dominated mixed strategy becomes extinct along every orbit.
- Appendix F. Omitted proofs from Section 4: No subset of the relative interior of a face is asymptotically stable under FTRL or steep-regularizer strategy flow.Every asymptotically stable set must intersect the vertex set A.
- F.1. Stability and attraction.: On one-dimensional faces, face-invariance and dominated-strategy elimination align the strategy flow with player deviations.The argument applies to any flow satisfying both structural properties.
- F.1. Stability and attraction.: An attracting asymptotically stable set must be closed under profitable deviations in its pure-profile skeleton.A strict better reply attracts trajectories along the connecting edge, while equal-payoff deviations make the edge stationary; either case contradicts exclusion from the set.
- F.1. Stability and attraction.: Every asymptotically stable set for the steep-regularizer strategy flow contains pure profiles closed under better replies.The result combines vertex intersection with the deviation-closure proposition.
- F.1. Stability and attraction.: Without face-invariance, large initial score gaps keep opponents near a profile while a better reply reverses the deviator’s choice and exits the stable neighborhood.The construction controls opponents over a time window, then makes the deviator’s score difference drift at a uniform negative rate.
- F.1. Stability and attraction.: The instability construction maintains each opponent within ρj of its initial pure action throughout [0, T].This is recorded as equation (F.5), while the deviator is arranged to dominate other actions by margin G′i at time T.
F.2. Connectedness. · Appendix G. Omitted proofs from Section 5 · G.1. Energy functions and asymptotic stability.
Strong connectivity of pure profiles yields internal chain transitivity of their mixed span, preventing proper attractors and implying dynamical connectedness. In parallel, local energy functions convert score-space dissipation into asymptotic stability under FTRL.
- F.2. Connectedness.: Strong connectivity of pure profiles propagates to internal chain transitivity of their mixed span under the strategy flow.For strongly connected H ⊆ A, span(H) is internally chain transitive.
- F.2. Connectedness.: The proof establishes span(H) invariance and rules out every proper attractor within the span.It uses invariant faces, induction over face dimension, and a contradiction involving an interior repellor.
- F.2. Connectedness.: Every attractor containing a strongly connected set H must contain all of span(H).The restricted attractor first contains all vertices of H, then all faces in span(H).
- F.2. Connectedness.: If the whole preference graph is strongly connected, the full strategy space X admits no proper attractor.This follows by applying the connectedness result with H = A.
- G.1. Energy functions and asymptotic stability.: A local energy function for a nonempty closed set S is Lipschitz and C1, vanishes exactly as Q(y) approaches S, and decreases strictly within sufficiently small positive energy bands.These conditions define the score-space dissipation mechanism used to establish stability.
- G.1. Energy functions and asymptotic stability.: Any nonempty closed set admitting a local energy function is asymptotically stable under FTRL.Energy-band decrease gives Lyapunov stability and eventual convergence of trajectories toward S.
- G.1. Energy functions and asymptotic stability.: The energy argument proves attraction by forcing E(t) → 0 and then using the equivalence Q(y) → S ⇔ E(y) → 0.If energy stayed above any positive threshold, its uniform negative derivative would eventually contradict nonnegativity.
G.2. When preferences are enough.
For subgames closed under better replies, preferences are sufficient to characterize asymptotic stability: their spans are attractors under the strategy flow, and attractors correspond exactly to such preference-closed sets. This also yields stability of singleton strict Nash equilibria and connects ordinal potentials with acyclic preference graphs.
- When preferences are enough: A subgame B closed under better replies has a Fenchel-gap local energy function for span(B), establishing its asymptotic stability.The construction verifies nonnegativity, vanishing exactly on the span, and dissipation along continuous-time FTRL.
- When preferences are enough: A span(B) is an attractor for the strategy flow if and only if B is closed under better replies.The forward implication follows from preference closure of attracting sets; the reverse implication is obtained by gluing facewise FTRL basins.
- Consequences: Every strict Nash equilibrium is a minimal attractor, while any minimal attractor in a finite game without ties must be a singleton strict Nash equilibrium.Better-reply paths from a closed set terminate at a strict Nash equilibrium contained in that set, forcing minimal attractors to be singletons.
- Consequences: An ordinal potential exists exactly when the preference graph is acyclic.An ordinal potential strictly increases along directed preference edges, while a topological ordering constructs one when no directed cycle exists.
G.3. When preference are not enough.
The counterexample shows that a preferentially stable set can have a dynamically unstable span under FTRL. Instability arises because trajectories initialized arbitrarily close to the set can be driven away, even from the interior and for every admissible regularizer.
- Counterexample: The counterexample is a three-player game in which each player has two actions, with profiles represented by three-letter words.The action sets are A1 = {F, B}, A2 = {L, R}, and A3 = {T, B}.
- Counterexample: At x∗ = (1/2, 1/2, 1), Player 3 has a strict profitable deviation from T to B, locally repelling trajectories from the top face.The restricted game on the top face has x∗ as a Nash equilibrium, but Player 3’s deviation drives instability.
- FTRL instability: The span of the unique club set is not stable under FTRL: every neighborhood of the set contains an admissible initialization whose trajectory eventually leaves a fixed neighborhood.The construction uses initial conditions converging to x∗ and proves finite-time escape from the fixed neighborhood.
- FTRL instability: The instability persists for full-support initializations converging to x∗, so escape occurs from the interior rather than from a proper subface.This distinguishes the example from instability arguments requiring initialization on a proper subface.
- Comparison with prior work: The instability argument applies to every FTRL dynamics satisfying the paper’s regularizer assumptions, whereas the cited prior result considered replicator dynamics only.The example is also consistent with the local-source instability result of Biggar and Papadimitriou.
Appendix H. Omitted proofs from Section 6
The appendix establishes mixed-profile characterizations of radness and constructs local energy functions proving that spans of strictly rad sets are asymptotically stable attractors under FTRL, including the entropic replicator case.
- Mixed-profile radness: Lemma H.1 characterizes radness through nonpositive payoff advantages on every face contained in span(H), with strict inequalities for s-radness.For every α outside H and face F ⊆ span(H), the condition is supx∈F Φ(x, α) ≤ 0, respectively < 0.
- Energy construction: For s-rad H with steep regularizer h, Lemma H.2 constructs ¯FH as a local energy function for span(H).The proof verifies regularity, positive semi-definiteness, and dissipativity, using strict radness to obtain a uniform negative bound on Φ near the span.
- Facewise stability: For every face F intersecting span(H), the restricted set SF = span(HF) is asymptotically stable under the corresponding facewise FTRL dynamics.The proof shows HF is strictly rad in the restricted subgame, constructs a facewise local energy function, and transfers stability to the strategy-flow restriction.
- Main theorem: Consequently, span(H) is a compact, invariant, asymptotically stable attractor for the strategy flow, and in the entropic case also for replicator dynamics.The strategy flow coincides with (RD) under the entropic regularizer.
- Stability consequence: The resulting energy function implies asymptotic stability of span(H) under the strategy flow.The construction supplies compactness, invariance, and attraction through facewise energy arguments and basin gluing.