CALM: Conditional Adversarial Latent Models for Directable Virtual Characters

Chen Tessler; Yoni Kasten; Yunrong Guo; Shie Mannor; Gal Chechik; Xue Bin Peng

Source-linked AI summary

CALM: Conditional Adversarial Latent Models for Directable Virtual Characters

Chen Tessler, Yoni Kasten, Yunrong Guo, Shie Mannor, Gal Chechik, Xue Bin Peng

arXiv:2305.02195v1May 2, 2023cs.CVcs.AIcs.RO

TL;DR

CALM addresses the challenge of generating diverse, realistic, and controllable behaviors for interactive virtual characters. It jointly learns a motion encoder and control policy from unlabeled motion data, then uses latent conditioning and high-level control to direct behavior. The reported results show semantic motion representations, directable policies, style-conditioned task control, and reuse on unseen tasks without further training.

Problem
Realistic virtual characters must generate diverse behaviors while responding naturally to environments and user inputs, but complex human motion makes this difficult.
Method
CALM jointly learns a motion encoder and low-level control policy from unlabeled motion data, then trains high-level control over encoded motion skills.
Results
CALM learns semantically meaningful motion latents and a directable policy, supports style-conditioned high-level control, and reuses both policies on unseen tasks without further training.
Takeaways & Limitations
Motion clips can serve as intuitive controls for physics-based characters and as guidance for higher-level task controllers.
Takeaways & Limitations
CALM targets in-distribution motions and cannot guarantee generated-motion quality for increasingly out-of-distribution unseen motions.

Abstract

from arXiv · show

In this work, we present Conditional Adversarial Latent Models (CALM), an approach for generating diverse and directable behaviors for user-controlled interactive virtual characters. Using imitation learning, CALM learns a representation of movement that captures the complexity and diversity of human motion, and enables direct control over character movements. The approach jointly learns a control policy and a motion encoder that reconstructs key characteristics of a given motion without merely replicating it. The results show that CALM learns a semantic motion representation, enabling control over the generated motions and style-conditioning for higher-level task training. Once trained, the character can be controlled using intuitive interfaces, akin to those found in video games.

1 INTRODUCTION

CALM addresses the challenge of generating realistic, diverse, and controllable behaviors by learning motion representations and policies from unlabeled human motion. It combines semantic skill encoding, directable control, and task reuse without further training.

Unlike independent controllers for individual clips, CALM targets scalable behavior generation, while prior latent-conditioned approaches could lose control over generated motion.
CALM learns a movement representation that captures diverse human motion while providing a directable interface for character control.
CALM’s policy need not precisely replicate reference clips, allowing natural behaviors that preserve motion characteristics while deviating from the data.
The method jointly trains a motion encoder and generative controller from unlabeled motion-capture data, directing motion through z = E(M).
Precision training reuses the pretrained policy and latent-space similarity to control motion during high-level tasks such as locomotion.
Combining the learned controller and precision training supports task solving without further retraining, using simple FSMs rather than meticulous reward and termination design.

2 RELATED WORK

Related work spans physics-based motion generation, direct task prediction, and latent-based control. CALM targets the shared limitation that learned latent spaces are difficult to control by learning dense semantic representations jointly with a directable policy.

Direct prediction solves predefined downstream tasks, whereas latent or language-based control learns generative motion models that can be directed.
Physics-based models generate motor actuations through simulation, enforcing dynamics such as gravity and friction during motion generation.
Latent-control methods learn behavior manifolds, but prior approaches commonly produce latent spaces that are complex to control.
CALM jointly learns a dense human-motion representation and a directable latent-conditioned policy to address this control difficulty.
Adversarial methods can generate motions likely under the reference distribution, supporting natural transitions between motion pairs not explicitly demonstrated.
ASE generated diverse behaviors but lacked global latent semantics, limiting easy motion-to-latent mapping and directability.

3 REINFORCEMENT LEARNING BACKGROUND

The paper formulates both pretraining and downstream control as reinforcement-learning problems in which policies interact with an environment over state transitions. The objective is discounted cumulative reward over trajectories.

At each timestep, the policy observes state s_t and samples action a_t, after which the environment transitions to s_t+1.
The reinforcement-learning objective maximizes discounted cumulative reward over trajectories generated by policy and environment transition probabilities.
The discount factor γ ∈ [0, 1) determines whether the agent emphasizes short-term or longer-term outcomes.

4 OVERVIEW

CALM builds directable controllers in three stages: low-level motion representation learning, high-level precision training, and inference without additional training. Encoded motions guide low-level behavior and higher-level task control.

CALM uses three stages: low-level encoder and generator training, high-level precision training, and inference without further training.
At inference, user-defined FSM rules select either direct low-level motions or high-level directed motions to compose complex behaviors.
During low-level training, encoder E maps reference motion M to a low-dimensional latent z, while a policy decoder generates similar motions in simulation.
The low-level policy produces varied behaviors on demand but initially lacks intuitive control over motion direction.
A high-level policy selects latents constrained near encoded reference motions, guiding behavioral style and movement direction.

5 CONDITIONAL ADVERSARIAL LATENT MODELS

CALM jointly learns a motion encoder and latent-conditioned policy from unlabeled motion data, using conditional imitation to produce directable and semantically structured motion representations.

CALM maps motions to latent variables and conditions both the policy and discriminator on those encodings to reproduce corresponding behaviors.This conditional setup is intended to mitigate mode collapse and preserve motion-specific correspondence.
The encoder is optimized through policy gradients, producing a latent space with semantic structure in which similar motions receive similar representations.
Two-second overlapping sub-motions provide coherent training units, while alignment and uniformity losses organize their predicted embeddings.Overlapping sub-motions come from the same sequence; randomly sampled sub-motions provide the independent comparison distribution.
Randomly changing the conditional motion during rollouts explicitly trains smooth transitions between disparate behaviors.
Gradient penalties and negative sampling improve adversarial training stability, while discriminator gradients are prevented from shaping the motion encoder.

6 HIGH-LEVEL CONTROL

CALM adds a high-level policy that combines task objectives with latent similarity to control both the form and direction of generated motions, enabling reusable task behaviors.

The high-level policy generates latent variables for the low-level controller while directing the character toward a specified motion and direction.
Training combines a task reward with a latent similarity loss so requested motion embeddings remain aligned with generated behavior.
A finite-state machine sequences motion demonstrations and supplies transitions or requested embeddings, allowing task descriptions to be specified naturally.
The same low-level policy, encoder, and high-level policy can solve unseen tasks in varying forms without retraining, resembling game-controller interaction.

.7 EXPERIMENTS

The experiments evaluate CALM’s pre-trained encoder and low-level policy using motion data, reporting measures of encoder quality, diversity, and controllability.

The low-level policy is trained on 160 motion clips totaling over 30 minutes, segmented into overlapping two-second sub-sequences.
Pre-training uses rollout-based interaction in which random reference motions are encoded and supplied to the low-level policy before PPO optimization.
Training runs across 4096 Isaac Gym environments for 5 billion steps, with low- and high-level decisions at 30 and 6 Hz, respectively.
Table 1 evaluates encoder quality, motion diversity, and generation controllability using Fisher’s concentration coefficient, Inception Score, and Generation accuracy.
The encoder is an MLP, while the policy and conditional discriminator parse latent inputs through additional heads before producing 31-dimensional actions.

8 RESULTS

CALM learns motion representations and controllers that support diverse, controllable behaviors, then reuses them for style-conditioned locomotion and zero-shot task solving.

8.1 Controllable motion generation: CALM learns semantically meaningful motion embeddings and a directable low-level controller, with improved encoder separation and generated-motion diversity.The evaluation measures encoder quality using Fisher’s class separability metric and diversity using Inception Score.
8.1 Controllable motion generation: Perceived generation accuracy increases from 35% to 78% compared with ASE, demonstrating improved control over requested motions.Raters judged whether generated motions matched reference motions and textual descriptions.
8.1 Controllable motion generation: CALM smoothly transitions between conditioned motions, including interpolation from sprint to crouching idle through progressively slower and lower movement.The interpolation follows a semantically meaningful path while maintaining continuous walking-like motion.
8.2 Solving downstream tasks: Precision training constrains latent codes near a reference encoding so a high-level policy preserves motion style while directing movement toward a requested heading.The high-level controller is trained on run, shield-raised walking, and crouch-walking styles.
8.2 Solving downstream tasks: Without further training or task-specific reward design, CALM composes motions to solve location and strike tasks in multiple forms.The system uses the directable low-level controller and high-level locomotion policy, with results shown qualitatively in Figure 5 and quantitatively in Table 2.

9 LIMITATIONS

CALM improves controllability but remains limited by mode collapse, out-of-distribution motions, intricate movement control, robustness to unfamiliar dynamics, and rendering artifacts.

Pre-training – mode collapse: CALM improves controllability over ASE from 35% to 78%, but mode collapse remains an open challenge.Idle-motion conditioning can produce unrealistic micromotions, so further improvement is needed.
Pre-training – unseen motions: The controller targets in-distribution motions and may fail as requested motions become increasingly out of distribution.Some unseen motions map to semantically similar learned motions, such as tip-toe mapping to bounce-walk.
Precision-training – beyond locomotion: Intricate movements such as sword or shield trajectories may require additional pre-training innovations and broader motion distributions.
FSM – robustness: The policy’s robustness envelope may limit tasks with dynamics unlike training, including stair climbing and uneven terrain.Further innovations and training may be required for such tasks.
Rendering – artifacts: Rendering in Omniverse can show penetrations and other artifacts because its visualization character differs geometrically from the physically constrained simulation character.The authors distinguish this rendering issue from an inherent CALM algorithm problem.

10 DISCUSSION AND FUTURE WORK

CALM jointly learns an unsupervised semantic motion representation and reusable controller, enabling directable skills, transitions, and higher-level task control across multiple settings.

10 DISCUSSION AND FUTURE WORK: CALM learns an encoder and low-level controller from unlabeled motions, mapping clips to semantic low-dimensional representations that generate similar-characteristic movements.The controller decodes latent motion representations into skills for a physically simulated character.
10 DISCUSSION AND FUTURE WORK: Future work targets disentangling motion direction from content and modeling coordinated motion-object interactions such as vaults and handsprings.The authors also note that motion-constrained training required delicate reward tuning.
10 DISCUSSION AND FUTURE WORK: Adversarial training encourages behavior likely under the reference distribution and enables novel transitions between motion classes beyond state-reconstruction generalization.
10 DISCUSSION AND FUTURE WORK: Unlike PADL, CALM requires no labels and jointly learns motion representation and control.
10 DISCUSSION AND FUTURE WORK: CALM’s latent space supports smooth semantic interpolation and a scalable controller that can solve multiple tasks.
10 DISCUSSION AND FUTURE WORK: The same low-level controller can be reused across tasks, while high-level policies provide latent variables that select desired motion characteristics.The approach supports task control through motion-conditioned locomotion and other task-specific observations.

D QUALITATIVE ANALYSIS OF THE LOW-LEVEL CONTROLLER

The qualitative analysis evaluates CALM’s learned representations, generation quality, transitions, and task composition, showing controllable motion generation and multi-step behavior without task-specific training.

D QUALITATIVE ANALYSIS OF THE LOW-LEVEL CONTROLLER: Human raters classify generated motions from CALM and ASE after conditioning on motion encodings, with mean accuracy reported across generated motions.The evaluation uses 40 reference motions and three generated motions per clip.
E.1 Ablation analysis: Adding negative samples and latent-space regularization improves generation quality in the pre-training ablation.The full model is compared against variants removing each design element.
D QUALITATIVE ANALYSIS OF THE LOW-LEVEL CONTROLLER: The low-level controller transitions between complex motions when successive reference clips provide corresponding latent conditions.
E.3.4 Strike: The strike task combines running toward a target with a sword strike while preventing unintended contact through its termination condition.
E.3 Qualitative task analysis: Iterative latent control produces new task-solving behaviors, including shield aiming for blocking and sword positioning for reaching.These behaviors extend beyond motions directly observed in the reference dataset.
E.3.6 FSM versus Reward design: FSM composition solves multi-step tasks without task-specific training by ordering requested motions, avoiding delicate reward and termination design.The approach can compose behaviors such as crouch-walking, kicking, and celebratory motion.

F LATENT SPACE ANALYSIS

CALM organizes motion classes into a semantically structured latent space, bringing similar motions closer together while preserving local similarity among subsequences.

The latent-space analysis evaluates five motion categories: walking, sword attacks, shield attacks, turning, and idle motions.
CALM clusters similar motion groups more closely in latent space than ASE, indicating stronger semantic structure.The comparison uses average pairwise distances between walking, sword attacks, shield attacks, turning, and idle motions; lower diagonal distances indicate tighter clustering.
Walking motions are dispersed across ASE’s latent space, showing that ASE does not maintain semantic relations between motion classes.
CALM maps subsequences from the same motion clip to nearby latent representations in the TSNE visualization.The TSNE plot includes approximately 180 motions, with only a subset labeled for clarity; TSNE does not preserve global structure.