PaperlayerView exact revision on arXiv

Source-linked AI summary

Game-Guided Skill Discovery through Self-Play for Playable Agent Control

Seungeun Rho, Jeonghwan Kim, Xue Bin Peng, Sehoon Ha

arXiv:2609.40137v1cs.LGcs.AIcs.RO

TL;DR

Skill discovery seeks compact motor skills that are semantically distinct and interpretable, while existing unsupervised methods may not ensure those properties together. GGSD uses hierarchical self-play against past policies to learn human-playable skills. Humans compose the learned skills to solve unseen tasks without additional training, though the repertoire is bounded by the behaviors required by the game.

  • Problem

    Skill discovery seeks compact, reusable skills, but unsupervised methods may yield distinct behaviors that are difficult to interpret or reuse as agent complexity grows.

    Source: §1 · p. 1

  • Method

    GGSD jointly learns a hierarchical policy through self-play, with a high-level policy selecting discrete skills and a skill-conditioned low-level policy producing motor actions.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    Humans compose GGSD skills to solve unseen locomotion and object-interaction tasks without additional policy training across Ant, Franka Arm, and Unitree G1 environments.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Game rules provide simple guidance for discovering diverse, reusable, human-playable skills.

    Source: §6 · p. 10

    The learned repertoire is bounded by game-required skills, so simple games may yield behaviors too narrow for some downstream tasks, such as dexterous manipulation after G1Boxing.

    Source: §6 · p. 10

Abstract

from arXiv · show

We present Game-Guided Skill Discovery (GGSD), a framework that uses self-play in games to discover motor skills that are directly playable by humans. Playable skills provide a compact abstraction for controlling embodied agents through a small set of learned behaviors rather than low-level actions. To be effective, these skills should be semantically distinct, interpretable, and expressive; properties that existing unsupervised skill-discovery methods often fail to achieve simultaneously. GGSD achieves these desiderata by grounding skill discovery in competitive gameplay. A hierarchical agent competes against its past selves, with a high-level policy selecting from a small discrete skill set and a skill-conditioned low-level policy learning the corresponding behaviors. After training, a human can replace the high-level policy and directly control the agent through the same discrete skills. Despite the small number of high-level actions, skill transitions give rise to emergent combo behaviors, expanding expressivity beyond individual primitives. Across Ant, Franka-arm, and Unitree G1 environments, we show that GGSD produces human-playable skills that humans can compose to solve unseen tasks, such as Maze and CubePush, without additional training. An interactive demo is available at https://ggsd-demo.github.io.

1 INTRODUCTION

GGSD uses competitive self-play to discover compact, human-playable motor skills, addressing the difficulty of obtaining skills that are both behaviorally meaningful and reusable. Humans can compose the learned skills to solve unseen tasks without further policy training.

1 INTRODUCTION

  • Existing unsupervised methods may produce distinguishable skills that are difficult to interpret or reuse, especially as agent complexity increases.Effective human-control skills should also be semantically distinct, interpretable, scalable to high-degree-of-freedom agents, and expressive despite a compact action space.Source: §1 · pp. 1–2
  • GGSD uses self-play in simple 1v1 games to learn a hierarchy whose high-level policy selects discrete skills and whose low-level policy maps them to motor actions.Game objectives guide useful behavior, while a mutual-information objective encourages distinct semantics across skill codes.Source: §1 · p. 2
  • After training, humans select skills through a small input vocabulary, while skill transitions produce emergent combos that expand expressivity beyond individual primitives.The high-level policy is replaced with human inputs, and the repertoire contains only five or six skills in each environment.Source: §1 · p. 2
  • Humans compose learned skills to solve unseen Maze locomotion and CubePush object-interaction tasks without additional policy training.The skills are learned across Ant, Franka Arm, and Unitree G1 environments.Source: §1 · p. 2

2 RELATED WORKS

Prior work shows that competition can induce motor behaviors, but this paper uses self-play itself to organize those behaviors into discrete skills directly controlled by humans. It contrasts game-rule guidance with language, demonstration, and reference-motion guidance.

2.1 SELF-PLAY FOR MOTOR SKILL LEARNING

  • A pool of historical opponents encourages skills that remain useful across opponents, and GGSD organizes competition-induced behavior into discrete, diverse, reusable skills for direct human control.Earlier self-play work also demonstrated motor behaviors such as running, blocking, tackling, and kicking; some high-DoF competitive work learned skills from reference motions first.Source: §2.1 · pp. 2–3

2.2 GUIDANCE IN SKILL DISCOVERY

  • Language descriptions, demonstrations, and reference motions provide external guidance, but can require impractical state descriptions or costly, repertoire-shaping motion datasets.GGSD instead uses lightweight game rules and self-play to discover behaviors beyond explicitly provided examples, including for high-DoF systems.Source: §2.2 · p. 3

3 PRELIMINARIES

The preliminaries define the single-agent MDP and then recast interaction with a fixed-policy opponent through induced dynamics. From the controlled agent’s perspective, a fixed opponent yields an ordinary MDP, while self-play changes that MDP as the opponent evolves.

3 PRELIMINARIES

  • An MDP is defined by state and action spaces, transition probabilities, rewards, and a discount factor; a policy maximizes expected discounted return.The notation is M = (S, A, P, r, γ).Source: §3 · p. 3
  • In a two-player game, the state combines the agent’s and opponent’s states, and each player takes actions; the controlled agent receives a game reward.The game dynamics depend on both players’ actions.Source: §3 · p. 3
  • For a fixed opponent policy, marginalizing its actions turns the game into an ordinary single-agent MDP; self-play repeatedly changes those dynamics as the opponent evolves.The overview depicts training against past checkpoints and subsequent human control of the high-level policy.Source: §3 · pp. 3–4, Figure 2 · p. 4

4 GAME-GUIDED SKILL DISCOVERY

GGSD turns simple competitive-game rules into human-playable skills through self-play and hierarchical learning. It trains against past opponents, co-learns strategy and motor behaviors, and makes a compact set of learned skills directly controllable by humans.

4.1 SELF-PLAY WITH A POOL OF PAST SELVES

  • Sampling historical opponents with extra weight on the latest checkpoint makes the self-play objective piecewise stationary while maintaining pressure from recent strategies.The latest checkpoint is sampled with probability p; the remaining probability is distributed uniformly across earlier checkpoints.Source: §4.1 · pp. 4–5

4.2 HIERARCHICAL SKILL LEARNING

  • The high-level policy selects a discrete skill every k steps, while the skill-conditioned low-level policy produces motor actions at every environment step.The policies are jointly trained: the high level optimizes the game objective, and the low level additionally receives a mutual-information reward.Source: §4.2 · p. 5

4.3 PLAYABLE LEARNED SKILLS

  • GGSD uses five or six discrete skills; high-level entropy encourages their use, while mutual information encourages distinct behaviors for different skill codes.The skill set is kept compact for human control, and state-dependent transitions can produce emergent combos.Source: §4.3 · p. 6
  • Restricting the low-level policy and discriminator to proprioception separates opponent-dependent strategy from motor skills, encouraging more consistent semantics for human control.The high-level policy can observe the full game state.Source: §4.3 · p. 6

5 EXPERIMENTS

GGSD learns competitive, semantically diverse skills and lets people use them without additional policy training. Human players achieved at least 84% average success across three unseen tasks, despite differences from the training games and simulator.

5.1 IMPLEMENTATION DETAILS

  • Experiments use AntSumo, AntFencing, FrankaAirHockey, and G1Boxing, with grouped opponent sampling across 4,096 parallel environments.Ten groups share sampled opponents, reducing the need to keep separate opponents for every environment.Source: §5.1 · pp. 6–7

5.2 GGSD LEARNS TO PLAY GAMES WELL

  • The final policy's combined win-and-draw rate remains above 50% against every evaluated checkpoint, while four users won only two of 20 AntSumo games.The win rate decreases and draw rate increases against more recent checkpoints; Figure 3 reports win and draw rates against the final policy's past checkpoints.Source: §5.2 · p. 7, Figure 3 · p. 7

5.3 QUALITATIVE ANALYSIS OF LEARNED SKILLS

  • Learned skills include interpretable turning, locomotion, and pushing in AntSumo, and locomotion, punching, and guarding in G1Boxing.Across seeds, behaviors important for gameplay, such as locomotion and striking, recur even when exact skill sets differ.Source: §5.3 · pp. 7–8

5.4 QUANTITATIVE ANALYSIS AGAINST BASELINES

  • GGSD achieves the highest semantic diversity across embodiments: approximately 2× the strongest baseline on Ant and G1, and 28% higher on Franka.Figure 6 presents semantic-diversity results, with higher values indicating greater diversity.Source: Figure 6 · p. 8, §5.4 · p. 9

5.3 QUALITATIVE ANALYSIS OF LEARNED SKILLS

  • Transitions between skills create purposeful combos: Franka transitions can strike the puck, while G1Boxing transitions can produce evasion or a strong forward punch.The effect of a skill can depend on the state induced by preceding skills, adding expressivity without increasing the number of human inputs.Source: §5.3 · p. 8

5.5 HUMAN PLAY ON UNSEEN DOWNSTREAM TASKS

  • At least 84% average success across all three unseen tasks shows that humans can compose learned skills without additional policy training, despite task and simulator shifts.The tasks include maze navigation and object manipulation; training used Isaac Lab and evaluation used MuJoCo.Source: Table 1 · p. 9, §5.5 · p. 9

6 CONCLUSION

The learned skills support interactive gameplay and can be reused by humans, without additional training, on unseen downstream tasks. The skill repertoire is bounded by the behaviors required by the game, which may constrain transfer.

6 CONCLUSION

  • Humans can reuse the learned skills without additional training to solve unseen downstream tasks.The skills also support interactive gameplay.Source: §6 · p. 9
  • Games provide simple guidance for discovering diverse, reusable, human-playable skills.Source: §6 · p. 10
  • The repertoire is bounded by the skills a game requires, so simple games may limit transfer to downstream tasks.A policy trained on G1Boxing is unlikely to acquire dexterous manipulation skills.Source: §6 · p. 10

A STATE-DEPENDENT SKILL SEMANTICS AND EMERGENT COMBOS

A compact skill set can gain expressivity through state-dependent transitions: the same skill may behave differently after the agent enters another state regime. Too many regimes, however, can make the interface difficult to understand.

A STATE-DEPENDENT SKILL SEMANTICS AND EMERGENT COMBOS

  • Skill transitions increase expressivity without adding buttons because a skill’s effect can depend on the state from which it is executed.One skill can move the agent into a state where another skill produces behavior absent when either is used alone.Source: A STATE-DEPENDENT SKILL SEMANTICS AND… · p. 13
  • Skill meanings may remain relatively predictable within behaviorally distinct state regimes, resembling a compact finite-state controller.Transitions between regimes let the same small set of inputs realize a broader behavioral repertoire.Source: A STATE-DEPENDENT SKILL SEMANTICS AND… · p. 13
  • In G1Boxing, a backward-leaning state lets a subsequent skill use stored posture and momentum for a substantially stronger forward punch.From ordinary standing states, skills instead produce behaviors such as walking, guarding, and punching.Source: A STATE-DEPENDENT SKILL SEMANTICS AND… · p. 13
  • The mutual-information discriminator distinguishes skill behaviors but does not require each skill to produce the same motion from every state.State-dependent effects can coexist with distinguishable semantics when behaviors remain sufficiently predictable and separable.Source: A STATE-DEPENDENT SKILL SEMANTICS AND… · p. 13
  • Too many context-dependent regimes could give the same button too many meanings and make the controller difficult to understand.The proposed middle ground is a small number of state-dependent regimes that increase compositional expressivity while retaining a compact, interpretable interface.Source: A STATE-DEPENDENT SKILL SEMANTICS AND… · p. 14

B AN ALTERNATIVE VIEW OF MUTUAL INFORMATION IN SKILL DISCOVERY

This section frames mutual information as organizing distinguishable behaviors into skill codes, while competitive games provide guidance about which behaviors are strategically useful. It also describes the game and observation setups for Ant Sumo and Ant Fencing.

B AN ALTERNATIVE VIEW OF MUTUAL INFORMATION IN SKILL DISCOVERY

  • Mutual information chiefly associates distinct behaviors with latent codes; it does not by itself ensure that behaviors are meaningful, useful, or natural.GGSD’s contribution is competitive-game guidance for useful behaviors, with MI organizing them into a discrete vocabulary for human control.Source: B AN ALTERNATIVE VIEW OF MUTUAL… · pp. 14–15
  • DIAYN uses entropy-driven exploration, ASE imitation, RGSD reference motions, and SDAX task rewards as additional sources of behavioral structure.These approaches combine their respective guidance with MI-based skill organization.Source: B AN ALTERNATIVE VIEW OF MUTUAL… · p. 14
  • GGSD’s self-play yields strategically meaningful behaviors such as punching, guarding, turning, or striking a puck because they serve winning.The game objective guides which behaviors are useful; MI encourages different codes to represent distinguishable behaviors.Source: B AN ALTERNATIVE VIEW OF MUTUAL… · pp. 14–15

C ENVIRONMENT DETAILS

  • Across the environments, both players share a policy under swapped roles; high-level policies and critics receive full game state, while low-level actors receive proprioception.The discriminator uses the low-level actor’s proprioception, except it omits previous-action dimensions in Ant Sumo and G1Boxing.Source: C ENVIRONMENT DETAILS · p. 15

C.1 ANT SUMO

C.2 ANT FENCING

  • Ant Fencing uses the Ant Sumo arena and agent configuration, with an additional loss when the opponent’s front foot contacts the torso with sufficient force.Its reward disables the push term; episodes last 10 seconds with random initial heading.Source: §C.2 · p. 17, Table 5 · p. 17

C.3 FRANKA AIR HOCKEY

Franka Air Hockey pits two 7-DoF arms against each other across a table, with the first to score two goals winning. The low-level actor observes the controlled arm, while puck, opponent, and match state are included only in the full state.

C.3 FRANKA AIR HOCKEY

  • Two 7-DoF Franka arms face each other across an air-hockey table; the first player to score two goals wins.Positions and velocities use each agent’s table frame, oriented toward the opponent’s goal, and the second player’s world x and y axes are negated.Source: §C.3 · p. 17
  • The low-level actor observes only the controlled arm, while puck, opponent, and match state are reserved for the full state.Table 6 describes observations; Table 7 lists reward terms and specifies a 12-second match, with the first to two goals winning.Source: §C.3 · p. 17, Table 6 · p. 17, Table 7 · p. 18

C.4 G1 BOXING

G1 Boxing pits two 23-joint humanoids against each other, with rounds decided by remaining hit points. The appendix also specifies the observation-space grouping, reward terms, and hierarchical-policy network architectures.

C.4 G1 BOXING

  • Two Unitree G1 humanoids with 23 actuated joints compete in a 10 m × 10 m arena; successful torso or head strikes reduce hit points, and higher remaining HP wins at timeout.A fall or arena exit voids the round; punch damage depends on glove closing velocity, with head hits multiplied by 1.5.Source: §C.4 · p. 18, Table 9 · p. 19
  • The G1 observation-space table distinguishes proprioceptive components from the complete observation set.Components marked with a check are included in sproprio; all components together form sall.Source: Table 8 · p. 18

D HYPERAPARAMETERS

  • For G1 Boxing, the high-level policy uses three hidden layers, while the low-level policy and critics use four and the discriminator uses two.The specified widths are [512, 512, 256] for the high-level policy, [512, 512, 256, 128] for the low-level policy and critics, and two 256-unit discriminator layers.Source: D HYPERAPARAMETERS · p. 20, Table 10 · p. 20

E VLM-BASED SKILL DIVERSITY METRIC

The metric evaluates skill diversity by captioning rendered skill rollouts and comparing their text embeddings. The section also specifies rollout and model settings, robot-specific prompts, and example captions.

E VLM-BASED SKILL DIVERSITY METRIC

  • Each policy contributes one reset-state, single-environment, 5-second rendered clip per skill: five skills for Ant and Franka, and six for G1.Clips are rendered at 1920 × 1080.Source: E VLM-BASED SKILL DIVERSITY METRIC · p. 20
  • The diversity score is the mean pairwise cosine distance among L2-normalized skill-caption embeddings; pairs containing an uninterpretable clip receive distance zero.Scores are averaged over three training seeds and reported as mean ± standard deviation.Source: E VLM-BASED SKILL DIVERSITY METRIC · pp. 20–21
  • A vision-language model captions sampled clip segments and a text-embedding model embeds the captions, using robot-specific time windows and fixed model settings.Ant uses 1–4 s, Franka 2–4 s, and G1 0–5 s; the sampled segment skips the initial settling phase.Source: E VLM-BASED SKILL DIVERSITY METRIC · pp. 20–21
  • The same prompt is used for every method on a given robot, asking about meaningful tasks for Ant and G1 but table-surface regions for Franka.The prompts instruct the model to return “no semantic meaning” for behaviors without a meaningful downstream task or interaction, with robot-specific wording.Source: E VLM-BASED SKILL DIVERSITY METRIC · p. 21
  • Example captions describe Ant self-righting, rotation, and crawling, a Franka arm stationary over the table’s bottom-right region, and G1 walking backward.These examples illustrate the semantic behavior descriptions produced for the clips.Source: E VLM-BASED SKILL DIVERSITY METRIC · p. 21

Related papers

Loading 2609.40137v1…