Source-linked AI summary
Synthesizing Physical Character-Scene Interactions
Mohamed Hassan, Yunrong Guo, Tingwu Wang, Michael Black, Sanja Fidler, Xue Bin Peng
TL;DR
Realistic character animation must account for interactions with objects, while many existing systems focus on simpler environments or require manually labeled motion data. This paper trains physically simulated characters with scene-conditioned adversarial imitation and reinforcement learning; it reports strong performance across sitting, lying down, and carrying, and generalization to varied objects and scenarios.
Problem
Many character-animation systems focus on simple, homogeneous environments with limited object interaction, and some scene-interaction methods require manual motion annotation.
Source: §1 · pp. 1–2Method
The method trains physically simulated characters with task rewards and a scene-conditioned discriminator that evaluates motion relative to objects, using unannotated motion data and randomized objects.
Source: §1 · p. 2Results
Takeaways & Limitations
Randomizing object properties and placements supports natural interactions across a wide variety of object shapes and placements beyond those depicted in the training data.
Source: §9 · pp. 7–8A separate policy is trained for each task, and some episodes end without the character completing the task.
Source: §8 · p. 7
Abstract
from arXiv · showhide
Movement is how people interact with and affect their environment. For realistic character animation, it is necessary to synthesize such interactions between virtual characters and their surroundings. Despite recent progress in character animation using machine learning, most systems focus on controlling an agent's movements in fairly simple and homogeneous environments, with limited interactions with other objects. Furthermore, many previous approaches that synthesize human-scene interactions require significant manual labeling of the training data. In contrast, we present a system that uses adversarial imitation learning and reinforcement learning to train physically-simulated characters that perform scene interaction tasks in a natural and life-like manner. Our method learns scene interaction behaviors from large unstructured motion datasets, without manual annotation of the motion data. These scene interactions are learned using an adversarial discriminator that evaluates the realism of a motion within the context of a scene. The key novelty involves conditioning both the discriminator and the policy networks on scene context. We demonstrate the effectiveness of our approach through three challenging scene interaction tasks: carrying, sitting, and lying down, which require coordination of a character's movements in relation to objects in the environment. Our policies learn to seamlessly transition between different behaviors like idling, walking, and sitting. By randomizing the properties of the objects and their placements during training, our method is able to generalize beyond the objects and scenarios depicted in the training dataset, producing natural character-scene interactions for a wide variety of object shapes and placements. The approach takes physics-based character motion generation a step closer to broad applicability.
1 INTRODUCTION
Realistic character animation must account for how objects constrain and afford movement, yet existing approaches have limits in interaction quality, generalization, or scalability. This work extends adversarial imitation learning to physically simulated scene interactions, using scene context and object randomization to support natural behavior across tasks and objects.
1 INTRODUCTION
- Scene interaction requires adapting movement to object-specific constraints and affordances, such as choosing different motions for a bunk bed and a sofa.Object size can also change the strategy required for picking an object up.Source: §1 · p. 2
- Existing approaches are limited in motion quality, generalization, or scalability, while tracking-based methods can struggle to compose and transition among diverse skills.The paper also notes that kinematic models can struggle to respect scene constraints in scenarios unlike their training environments.Source: §1 · p. 2
- The method extends adversarial motion priors with a discriminator that evaluates character motion together with scene objects, without manually annotating motion data.The policy balances task reward with motion realism in context; the approach also randomizes object properties and configurations during training.Source: §1 · p. 2
- The framework demonstrates sitting, lying down, and carrying, with policies that transition between locomotion and task-specific behaviors.The authors report superior performance to prior state-of-the-art kinematic and physics-based methods.Source: §1 · p. 2
2 RELATED WORK
Prior character-scene interaction methods include motion editing, learned kinematic models, and physics-based controllers. The paper positions its approach as non-hierarchical, applicable to multiple tasks, and able to generalize across objects and scenes without manual motion labels.
2 RELATED WORK
- Traditional animation edits, retargets, or replays clips, but such methods are limited to scenarios similar to the source motion.The cited approaches adapt motions to new characters or environments, including scene interactions such as carrying boxes and climbing ladders.Source: §2 · p. 2
2.1 Deep Learning Kinematic Methods
- Deep-learning kinematic methods synthesize interactions such as sitting and carrying, but typically need scarce high-quality 3D human-scene data and degrade on unfamiliar scenarios.Some methods use phase variables, mixtures of experts, or automatically computed local motion phases.Source: §2.1 · pp. 2–3
2.2 Physics-Based Methods
- Physics-based methods ensure physical plausibility, but simulated bodies may not produce life-like behavior, and tracking one reference at a time complicates composing diverse skills.Adversarial motion priors can imitate and compose unannotated behaviors from large datasets.Source: §2.2 · p. 3
2.3 Character-Scene Interaction
- Earlier physical interaction methods include attached-object manipulation and hierarchical controllers; this approach instead is non-hierarchical and generalizes to more objects and scenes.Its discriminator evaluates motion in context using information about both the character and the environment.Source: §2.3 · p. 3, Figure 2 · p. 3
3 METHOD
The method trains a physically simulated character with a task reward and an adversarial style reward. Its discriminator compares policy behavior with motion-dataset behavior, while the policy maximizes expected discounted return.
3 METHOD
- The policy combines task and style rewards to pursue a goal while imitating dataset behaviors, and is trained to maximize expected discounted return.Task rewards cover objectives such as sitting or moving an object; style reward encourages realistic dataset-like behavior during the task.Source: §3 · pp. 3–4
- An adversarial discriminator models the style reward by comparing simulated-character motions with motions from a dataset.Its objective includes a manually specified gradient-penalty coefficient.Source: §3 · p. 4
4 STATE AND ACTION REPRESENTATION
The state representation includes character and object configuration, allowing the discriminator to assess motion relative to objects. The policy outputs joint targets for PD controllers, and separate policies address the three tasks.
4 STATE AND ACTION REPRESENTATION
- The state combines character-body configuration with object configuration relative to the character, yielding a 114D feature space.Features include root and joint information, key-joint positions, and object position and orientation.Source: §4 · p. 4
- Including object features lets the discriminator judge motion realism relative to the object, which may move and require the agent to react.Root height and rotation use world coordinates, while root velocities and object-related positions and orientation use character-local coordinates.Source: §4 · p. 4
- The policy action specifies joint target rotations for PD controllers, represented as exponential maps in a 28D action space.Separate policies are trained for sitting, lying down, and carrying, using the same style reward.Source: §4 · p. 4
5 MOTION DATASET
The motion data combines human demonstrations of sitting, lying down, and carrying with training-time exposure to many object instances and configurations.
5 MOTION DATASET
- Sitting and lying demonstrations come from 100 minutes of SAMP motion-capture clips covering seven objects, while carrying uses 15 clips of a subject moving one box.The carry clips vary initial and target box locations and track the box with optical markers.Source: §5 · p. 4
- Training expands beyond the demonstrated objects using about 350 ShapeNet instances, random object scaling, and varied object configurations.The object categories include chairs, tables, stools, sofas, and beds; carry boxes are scaled by factors from 0.5 to 1.5.Source: §5 · p. 4
- The demonstrations cover only seven scene objects and one carry box, but the training environments expose policies to additional object instances.Different objects are placed in parallel simulated environments to encourage interaction with natural class variation.Source: §5 · p. 4
6 TRAINING
Training varies episode initial states and object configurations so policies learn to approach targets and perform tasks across scene setups.
6 TRAINING
- Characters and objects start from randomly sampled dataset states, ranging from far-from-target starts to near-completion states.Reference State Initialization is described as speeding training and producing more realistic motions than a fixed start state.Source: §6 · p. 4
- Object positions are randomized from one to ten meters away, with orientations sampled uniformly from 0 to 2π, to cover configurations absent from reference motions.Episodes last 10 seconds for sitting and lying down and 15 seconds for carrying.Source: §6 · pp. 4–5
- The policy network maps the current state and goal to the mean of a Gaussian action distribution with manually specified, fixed covariance.The policy, value function, and discriminator use separate fully connected networks and are trained jointly.Source: §6 · p. 5
7 RESULTS
The results show successful scene interactions across sitting, lying down, and carrying, including transitions from locomotion to task-specific behavior. Evaluation reports broad object and configuration generalization, robustness, and comparisons with prior systems.
7 RESULTS
- Characters approach objects from random orientations, then transition from walking or running into sitting, lying down, or picking up an object.In carrying, the box is a simulated rigid body moved by forces rather than attached to the character’s hand.Source: §7 · p. 5
- Policies sit and lie down on 21 unseen objects after training on about 350 objects, and carry boxes sampled from 25 × 17.5 × 15 cm to 75 × 52.5 × 45 cm.The demonstrations involved seven objects for sitting and lying down and a single box for carrying.Source: §7 · p. 5
- Characters display varied sitting styles on the same object, including regular sitting, leaning backward, and differing arm movements.Source: §7 · p. 5, Figure 7 · p. 6
7.1 Evaluation
- Policies exceed 90% success across tasks despite diverse test objects and configurations, with metrics averaged over 4,096 trials per task.The reported metrics are success rate, execution time, and precision.Source: §7.1 · p. 5, Table 1 · p. 5
- Policies execute tasks from starts up to ten meters away, although reference clips start no more than three meters from the object.The authors attribute this generalization partly to scene randomization during training.Source: §7.1 · p. 5
- Policies recover and resume tasks after projectile impacts, and maintain high success under physical perturbations across all three tasks.The authors also test sudden object movement during task execution.Source: §7.1 · pp. 5–6
7.2 Comparisons
- Compared methods exhibit footskating, floating, or object penetration, while the physical approach of Chao et al. often falls before sitting successfully.Source: §7.2 · p. 7
- The method significantly outperforms prior systems on sitting and lying-down tasks.The comparison includes NSM, SAMP, and Chao et al.’s physical approach.Source: §7.2 · p. 7, Table 3 · p. 7
8 DISCUSSION
The discussion identifies per-task training, occasional task failures, and restricted scene scope as limitations, while noting generalization across objects and physical properties within tested ranges.
8 DISCUSSION
- The experiments train a separate policy for each task, leaving multi-task reinforcement learning for future work.The authors describe multi-task RL as difficult and open.Source: §8 · p. 7
- Unlike earlier carry methods, the box is a rigid simulated object moved by the character’s applied forces rather than welded to its hand.The authors contrast this setup with Coros et al. 2010, Mordatch et al. 2012, and Peng et al. 2019.Source: §8 · p. 7
- Some episodes end with the character beside the object or short of the target, sometimes after following a suboptimal path.The authors give standing beside the object and failing to reach it in time as examples.Source: §8 · p. 7
- The experiments use single-object environments; the authors suggest extending the state representation and adding virtual eyes for more complex scenes.The state representation could be augmented to contain other objects.Source: §8 · p. 7
- Randomization supports interaction with unseen ShapeNet test objects and varied sizes and weights, but success is expected to drop when test values are far outside training ranges.The authors also leave generalization to different skills, such as jumping, for future work.Source: §8 · p. 7
9 CONCLUSION
The conclusion presents a scene-conditioned policy and discriminator for physical character-scene interaction, trained with randomized objects and configurations. Across three tasks, the policies transition between behaviors and generalize to varied objects and unseen scenarios.
9 CONCLUSION
- The method introduces a scene-conditioned policy and discriminator that account for character movement in the context of environmental objects.The conclusion applies the method to sitting, lying down, and carrying, and says it learns when and where to transition between behaviors.Source: §9 · p. 7
- Randomizing training objects, placements, sizes, and physical properties allows policies to generalize to a wide range of objects and scenarios absent from demonstrations.The policies are also reported to be robust to physical perturbations and sudden environmental changes.Source: §9 · pp. 7–8
- The authors report that their method significantly outperforms previous systems in qualitative and quantitative evaluations.Source: §9 · p. 8
- The authors frame the system as a step toward physically simulated characters that interact with complex environments in a more intelligent and life-like manner.Source: §9 · p. 8
Appendices
The appendix defines task goals and rewards for sitting, lying down, and carrying. Each task directs the character toward an object or destination, with near-target rewards specifying the desired interaction.
A TASKS
- The task set comprises sitting, lying down, and carrying, using a shared style reward and task-specific goal reward.The sit task places an object at a random orientation one to ten meters from the character.Source: A TASKS · p. 10, §A.1 · p. 10
A.1 Sit
- For sitting, the goal is the object’s bounding box; rewards encourage walking toward the object and then placing the character’s hip at a target sitting position.The target walking speed is 1.5 m/s.Source: §A.1 · p. 10
A.2 Lie down
- For lying down, the character walks to an object and lies on it, using the sitting task’s goal and far reward while the near reward uses head height.The prose identifies character head height and target head height.Source: §A.2 · p. 10
A.3 Carry
- For carrying, the character picks up a box and moves it to a destination; rewards guide the approach, transport, and precise placement on the target platform.The reward descriptions also refer to a local-frame target and the box’s height, width, and depth.Source: §A.3 · p. 10
B RESULTS
The results show that object bounding-box information supports successful scene interaction, while policies produce plausible trajectories and adapt carrying behavior to box weights beyond the single weight demonstrated.
B RESULTS
- A carrying policy trained from a demonstration of one box adapts to boxes weighing 5–26 kg by conditioning its goal on box density.The demonstrated carrying style remains constrained by the single style in the reference data; learning other styles requires those styles in the dataset.Source: B RESULTS · p. 10
- Without object bounding-box information, success rates decrease across all three tasks, and the character fails to pick up the box in the carry task.The table reports success rate, average execution time, and average precision for all tasks.Source: Table S.1 · p. 11, B RESULTS · p. 11
- Initialized from reference clips, policies often take a shorter path to the object than the reference, with trajectories compared for carry, sit, and lie-down tasks.The carry comparison plots the box trajectory; sit and lie-down comparisons plot the character trajectory.Source: B RESULTS · pp. 10–11, Figure S.2 · p. 11
C COMPARISONS
- The comparison evaluates goal-driven NSM and SAMP against the method with the same user inputs: desired action and target object.The full motion is generated by each algorithm, without manual or random control.Source: C COMPARISONS · p. 11