PaperlayerView exact revision on arXiv

Source-linked AI summary

Hierarchical Planning and Control for Box Loco-Manipulation

Zhaoming Xie, Jonathan Tseng, Sebastian Starke, Michiel van de Panne, C. Karen Liu

arXiv:2306.09532v2cs.ROcs.GR

TL;DR

Cluttered box rearrangement calls for coordinated locomotion and manipulation, which remain challenging to synthesize in animation. This paper uses hierarchical planning, diffusion-generated locomotion references, and physics-based imitation policies to control a simulated character. It demonstrates box rearrangement behaviors and identifies discrete controllers and their transitions as a current limitation.

  • Problem

    Cluttered rearrangement remains difficult to synthesize because characters must navigate obstacles while coordinating locomotion and manipulation.

    Source: §1 · p. 2

  • Method

    A hierarchical controller combines planning, waypoint-conditioned diffusion locomotion references, and object-aware physics-based imitation policies.

    Source: §1 · p. 2

  • Results

    The system demonstrates simulated characters fetching, carrying, placing, and stacking objects, and diffusion motions that RL imitation controllers can track directly.

    Source: §8 · p. 16

  • Takeaways & Limitations

    The demonstrations show diffusion models can provide mid-level plans of sufficient quality for direct tracking by RL-based online imitation controllers.

    Source: §8 · p. 16

    The system currently uses discrete controllers, and transitions between manipulation and locomotion remain a limitation.

    Source: §8 · p. 16

Abstract

from arXiv · show

Humans perform everyday tasks using a combination of locomotion and manipulation skills. Building a system that can handle both skills is essential to creating virtual humans. We present a physically-simulated human capable of solving box rearrangement tasks, which requires a combination of both skills. We propose a hierarchical control architecture, where each level solves the task at a different level of abstraction, and the result is a physics-based simulated virtual human capable of rearranging boxes in a cluttered environment. The control architecture integrates a planner, diffusion models, and physics-based motion imitation of sparse motion clips using deep reinforcement learning. Boxes can vary in size, weight, shape, and placement height. Code and trained control policies are provided.

1 INTRODUCTION

Box rearrangement in cluttered environments remains challenging because it requires navigating obstacles while coordinating locomotion and manipulation. The paper combines hierarchical planning, diffusion-generated locomotion references, and physics-based imitation policies to perform these tasks.

1 INTRODUCTION

  • Cluttered box rearrangement requires characters to navigate obstacles and coordinate locomotion with manipulation.The system demonstrates lifting, carrying, and placing boxes in environments with obstacles.Source: §1 · p. 2
  • A hierarchy separates spatial planning from motor-skill execution, allowing locomotion and manipulation skills to generalize across arrangement tasks.A high-level planner establishes a path between pick-up and place-down locations.Source: §1 · p. 2
  • Diffusion generates waypoint-conditioned locomotion trajectories, while a single motion clip serves as the reference for box pickup and placement.The paper introduces a bidirectional root representation to improve waypoint satisfaction; manipulation motion data is more limited.Source: §1 · p. 2
  • Object-aware imitation-based reinforcement-learning policies track reference motions and support carrying boxes across varied weights, sizes, and pick-up or place-down heights.The low-level policies provide fully physically simulated motor control.Source: §1 · p. 2
  • The authors demonstrate component robustness and conduct ablations, with contributions spanning hierarchical control, bidirectional-root diffusion planning, and generalized lift-and-place skills.The reported generalization spans box weights and heights.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses locomotion, manipulation, and motion generation through separate kinematic or physics-based approaches, while the paper situates its work among efforts to combine those capabilities. The related work emphasizes limitations in environmental interaction and the challenges of coordinating skills.

2 RELATED WORK

  • Prior work often focuses on locomotion or manipulation separately, although humans fluently combine both skills.The paper frames their combination as difficult to reproduce in animation.Source: §2 · p. 2

2.1 Loco-manipulation for Humanoids

  • Kinematic loco-manipulation can produce high-quality motion but may lack interaction detail and weight; physics-based methods face simplified interactions, compute demands, or motion-quality limits.The authors contrast their system with prior approaches by highlighting obstacle avoidance, diffusion-driven motion tracking, and richer object interaction.Source: §2.1 · pp. 2–3

2.2 Reinforcement Learning for Character Animation

2.3 Kinematic Motion Generation for Character Animation

  • Kinematic motion generation uses motion-capture queries, supervised learning, and generative models, including diffusion, to synthesize motion from input.The cited generative approaches include variational autoencoders and diffusion models.Source: §2.3 · p. 4

3 SYSTEM OVERVIEW

The system represents box rearrangement as a sequence of four motion primitives connected by a simple graph. A planner supplies waypoints, and each primitive generates or retrieves reference motion for a physics-based policy to track.

3 SYSTEM OVERVIEW

  • Four primitives—walk-only, walk-and-carry, pick-up, and put-down—are sequenced according to a graph of permitted transitions.An arrow from A to B means B can follow A, but not vice versa.Source: §3 · p. 4
  • For each locomotion primitive, the high-level planner generates root-position and heading waypoints to navigate toward task locations while avoiding scene collisions.The planner operates on a 3D scene and a primitive path.Source: §3 · p. 4
  • Locomotion uses diffusion-generated walking references trained on about 25 minutes of motion capture, while pick-up and put-down use one reference trajectory with environment-conditioned policies.The policies track the kinematic references through physics simulation; box motion capture could make manipulation motion more natural.Source: §3 · p. 4

4 LOCOMOTION

The locomotion system has walk-only and walk-and-carry primitives. Both use a shared diffusion model for waypoint-conditioned whole-body reference trajectories, with separate reinforcement-learning policies for tracking.

4 LOCOMOTION

  • Both primitives use the same diffusion model to generate whole-body reference trajectories conditioned on waypoints.The resulting kinematic motions guide the low-level controllers.Source: §4 · p. 4
  • Each locomotion primitive has its own low-level reinforcement-learning policy to track the shared model’s reference trajectory.The two policies correspond to walk-only and walk-and-carry.Source: §4 · p. 4
  • The locomotion system defines two primitives: walk-only and walk-and-carry.Figure 3 shows walk-only at the top and walk-and-carry at the bottom.Source: §4 · p. 4, Figure 3 · p. 5

4.1 Diffusion-based Whole-body Motion Generator

The motion generator uses a diffusion model to produce whole-body navigation sequences conditioned on waypoints. Bidirectional root control and continuity between short sequences support waypoint tracking and long-form motion generation.

4.1 Diffusion-based Whole-body Motion Generator

  • The diffusion model generates whole-body navigation sequences by iteratively denoising noise, learning to reconstruct motion data from noisy inputs.It is trained on the locomotion dataset from.Source: §4.1 · p. 5

4.1.1 Diffusion Process.

  • The model learns to reverse a Markov noising process by estimating the original motion from noisy latents conditioned on target trajectories.The paper follows the diffusion definition in [Ho et al. 2020].Source: §4.1.1 · p. 5

4.1.2 Kinematic Motion Representation.

  • Waypoint conditioning specifies two target positions and directions, while the model predicts root trajectories and joint angles over 2N frames.The waypoints are reached at frames N and 2N; N=15 spaces them 0.5 seconds apart.Source: §4.1.2 · pp. 5–6

4.1.3 Bidirectional Root Control.

  • A weighted average of root trajectories represented relative to both the start pose and waypoint is used to eliminate drift; egocentric-only prediction causes discontinuities.The weighting decreases to zero as the character approaches the waypoint over N frames.Source: §4.1.3 · p. 6

4.1.5 Parallel Long-form Generation.

  • Training on two waypoints at a time and matching each sequence’s first waypoint to the previous sequence’s second produces globally consistent long motions.The approach extends parallel long-form generation from [Tseng et al. 2022] to tracking arbitrarily long trajectories.Source: §4.1.5 · pp. 6–7, Figure 5 · p. 7

4.2 Locomotion Control Policy

Physics-based policies track reference motions generated or sampled for locomotion and carrying. Reinforcement learning, imitation rewards, and object-specific rewards support walking and box transport.

4.2 Locomotion Control Policy

  • Reinforcement-learned policies track whole-body reference trajectories under physics simulation, using the same motion data as the diffusion model.The system uses RAISIM and PPO, simulating at 120 Hz and querying the policy at 60 Hz.Source: §4.2 · p. 7

4.2.1 Policy Setup.

  • The policy combines character state and reference-frame deviations as inputs, with multiplicative imitation rewards, PD-plus-torque actions, and output smoothing.Multiplying joint, translation, and orientation rewards prevents the policy from ignoring any reward term; smoothing addresses high-frequency torque fluctuations associated with ankle vibration and foot sliding.Source: §4.2.1 · pp. 7–8

4.2.2 Walk-only Policy.

  • The walk-only policy imitates motion-capture sequences of human walking in different directions and at different speeds.Episodes begin from the first frame of a randomly sampled sequence.Source: §4.2.2 · p. 8

4.2.3 Walk-and-carry Policy.

  • The walk-and-carry policy adds box position and orientation inputs and rewards hand placement on opposing sides, upright orientation, and alignment between stacked boxes.Training randomizes the number of boxes carried, adding orientation and stack-alignment rewards for each additional box.Source: §4.2.3 · p. 8

5 MANIPULATION

The manipulation section defines shared Pick-up and Put-down primitives and their policy setup, then describes how locomotion and manipulation actions are coordinated at transitions.

5 MANIPULATION

  • Pick-up and Put-down share one reference trajectory and one low-level policy.Source: §5 · p. 9

5.1 Manipulation Motion

  • The shared reference uses crouching and standing, with arm transitions distinguishing picking up a box from putting it down.For pick-up, the arms move from rest to carrying pose while crouching; for put-down, they return to rest while standing.Source: §5.1 · p. 9

5.2 Manipulation Policy

  • Training repeats pick-up and put-down from randomized platform heights; simultaneous training exposes the unified policy to varied relative states.The unified policy is intended to support varied boxes and repeated task concatenation for long-horizon animation.Source: §5.2 · p. 9

5.2.1 Policy Setup.

  • Policy inputs include platform placement, a one-hot pick-up/put-down flag, and locomotion-policy inputs.Source: §5.2.1 · p. 9
  • The target box height interpolates from platform height to the character’s 1.2 m torso height for pick-up, with the direction reversed for put-down.Source: §5.2.1 · pp. 9–10
  • The per-step reward adds r_base outside the product of joint, translation, and orientation rewards, allowing trade-offs between manipulation goals and reference tracking.The policy is also trained to manipulate multiple boxes.Source: §5.2.1 · p. 10

5.3 Transition Between Locomotion and Manipulation

  • Transitions add 30–60 standing-pose frames: locomotion is brought to a stop before manipulation, and actions are linearly blended when returning to locomotion.Source: §5.3 · p. 10

6 PLANNER

The planner uses kinodynamic A* to route between boxes while accounting for obstacles and realistic navigation dynamics, then smooths paths for diffusion-model conditioning.

6 PLANNER

  • Kinodynamic A* generates obstacle-avoiding paths between boxes using a waypoint per box location and desired pick-up orientation, while maintaining realistic velocities and accelerations.Source: §6 · p. 10
  • Cubic splines smooth each planned trajectory to maintain reasonable linear and angular velocities.Source: §6 · p. 10
  • Goal conditioning for the diffusion model is sampled from the smoothed path; the authors describe A* as replaceable by alternatives such as D* or RRT.Source: §6 · p. 10

7 RESULTS AND DISCUSSION

The system demonstrates box rearrangement through diverse long-horizon tasks and tests the robustness of its manipulation and locomotion components. Its bidirectional root-control model reduces average trajectory error, while the hierarchical design has limitations around feedback and transitions between primitives.

7.1 Diversity of Rearranging Tasks

  • Two long-horizon animations show the system rearranging assorted boxes in two scenes using four motion primitives.The system also demonstrates rearranging cylinders and playing Tower of Hanoi.Source: §7.1 · pp. 10–11

7.2 Generalizability of Manipulation Policy

  • Manipulation tests vary pickup and placement locations, box weight, and box width; the policy’s tested extremes are recorded alongside its training distribution.Parameters are varied one at a time, with success requiring at least three completed pickup-and-put-down trials.Source: §7.2 · p. 11, Table 1 · p. 11

7.3 Robustness of Locomotion

  • The locomotion policy tracks all 10 randomly waypoint-conditioned, 10-second diffusion trajectories despite independent training of the model and policy.The walk-and-carry policy also withstands heavy projectiles while holding boxes; box width and weight are tested separately.Source: §7.3 · pp. 11, 14, Table 2 · p. 14

7.4 Comparison To DReCon

  • Tracking performance is higher for diffusion-generated trajectories than for motion-matching trajectories, although the controller was never trained on diffusion-generated data.The authors suggest the diffusion output is more in-distribution with motion-capture data; motion matching requires storing a queryable motion database.Source: §7.4 · p. 14, Table 3 · p. 15

7.5 Ablation on Bidirectional Root Control

  • The bidirectional model reduces average reference-trajectory error by 70% and avoids the egocentric model’s poor long-tail behavior.The models perform similarly in simpler cases, but the egocentric model deviates visibly during precise turns.Source: §7.5 · p. 15, Figure 11 · p. 15

7.6 Comparison to Other Hierarchical Framework.

  • The system’s planner enables rapid task design, but the architecture has no feedback between the mid-level motion generator and low-level controller.Root tracking errors can cause task failure in clutter, and predefined primitive paths can produce less natural transitions, including a required standing pose before manipulation.Source: §7.6 · pp. 15–16

8 CONCLUSIONS

The system demonstrates physically simulated characters fetching, carrying, placing, and stacking objects, with diffusion-generated motions tracked by RL-based imitation controllers. Its current discrete controllers limit transitions between manipulation and locomotion, and heavier-object handling and greater motion fidelity remain future directions.

8 CONCLUSIONS

  • Physically simulated characters fetch, carry, place, and stack objects, while diffusion-generated motions can be directly tracked by RL-based online imitation controllers.The work also demonstrates adding object interactions to low-level RL-driven physics-based imitation methods.Source: §8 · p. 16
  • Discrete controllers limit transitions between manipulation and locomotion; a unified diffusion model and low-level controller are proposed as future work.The authors also identify dynamic replanning and obstacle avoidance as possible extensions to improve robustness.Source: §8 · p. 16
  • Handling heavier boxes may require grasping fingers or handles and full-body support, while gaze, biomechanical models, and richer manipulation controllers could improve human-motion fidelity.Source: §8 · p. 16

Related papers

Loading 2306.09532v2…