PaperlayerView exact revision on arXiv

Source-linked AI summary

InstantMimic: A High Performance System for Learning Physics-based Skills in Seconds

Ikjun Choi, Geonho Leem, Jungdam Won

arXiv:2609.09821v1cs.GRcs.RO

TL;DR

Physics-based Deep RL control achieves realistic skills but remains inefficient because GPU-accelerated pipelines underutilize hardware around the physics solver. InstantMimic makes the full training loop GPU-native, reducing diverse skill-learning tasks to seconds and enabling practical LLM-driven hyperparameter optimization.

  • Problem

    GPU-accelerated physics does not ensure efficient end-to-end training because fragmented kernels and CPU memory access underutilize the hardware.

  • Method

    InstantMimic integrates simulation, environment computation, policy inference, and policy updates into a unified GPU-native execution flow.

  • Results

    Training standard motion-tracking skills takes seconds, while latent-controller pretraining on AMASS takes 30 minutes.

  • Takeaways & Limitations

    Seconds-level training makes automated experimentation practical, including LLM-agent search that iteratively refines configurations.

  • Takeaways & Limitations

    HPO can favor faster convergence over tracking quality because candidate ranking directly uses training time to success.

Abstract

from arXiv · show

Physics-based character control is a long-standing challenge in computer graphics and robotics, requiring policies that satisfy complex dynamics while producing realistic motion. Recent Deep RL approaches, particularly imitation learning methods such as DeepMimic, have had broad impact beyond animation, influencing robotics by enabling agile and expressive behaviors. While these approaches achieve impressive results, they remain computationally inefficient to train in practice. Despite GPU-accelerated simulation, we find that end-to-end pipelines often underutilize hardware due to overheads outside the physics solver, caused by fragmented GPU kernels and CPU memory access in the critical path. We present InstantMimic, a system that addresses these inefficiencies by making the entire training loop GPU-native. Built on a GPU-native physics backend, our unified pipeline integrates simulation, environment computation, policy inference, and policy updates within a single execution flow. As a result, InstantMimic reduces training time for diverse physics-based skills to a few seconds and makes LLM-agent-driven hyperparameter search practical.

1 Introduction

Physics-based control offers realistic, generalizable motion but remains difficult and costly to train. InstantMimic makes the full Deep RL training loop GPU-native, reducing skill-learning time to seconds and supporting automated experimentation.

  • Physics-based character control must satisfy complex dynamic constraints while producing realistic motion.
  • DeepMimic-style imitation learning enables neural policies to reproduce locomotion, acrobatics, and transitions from reference motions.
  • GPU-accelerated simulation still leaves training inefficient because fragmented kernels and CPU memory access underutilize hardware.
  • InstantMimic integrates simulation, environment computation, and policy updates in a unified GPU-native training loop.The design minimizes kernel-launch overhead and avoids implicit synchronization through batching and fusion.
  • Training diverse physics-based skills takes only a few seconds, making LLM-agent hyperparameter search practical.The contribution list describes seconds-level learning and automated experimentation with minimal human intervention.

2 Related Work

Prior work establishes GPU acceleration, GPU-computing optimizations, and automated hyperparameter search as foundations for efficient Deep RL. InstantMimic extends these ideas by integrating simulation, inference, and PPO optimization for high-DoF motion imitation.

  • 2.2 Optimization Principles in GPU Computing: Kernel fusion and CUDA Graphs reduce GPU launch overhead through compile-time merging and runtime replay, respectively.
  • 2.2 Optimization Principles in GPU Computing: Keeping rollout state and training data on the device avoids costly movement across the CPU–GPU boundary and between training stages.
  • InstantMimic integrates simulation, policy inference, and PPO optimization into one GPU-native framework for high-DoF humanoid motion imitation.This contrasts with frameworks that leave policy inference and optimization external to environment simulation.
  • 2.3 Hyperparameter Optimization for Deep RL: Existing HPO methods require rapidly increasing evaluations as search dimensionality grows, encouraging small hand-picked search spaces.
  • 2.3 Hyperparameter Optimization for Deep RL: Search-space design remains largely dependent on human expertise, while LLM-based methods can explore configurations more extensively through automation.

3 Profiling Bottlenecks in the Baseline

Profiling shows that the baseline wastes most rollout time outside physics simulation. Fragmented post-physics kernels and CPU-mediated memory access create stalls that keep the GPU underutilized.

  • The baseline GPU is stalled or underutilized for 77% of wall time per rollout step despite using a GPU-based physics engine.
  • Fragmented post-physics kernels and CPU memory access are the two root causes of the observed bottlenecks.Python-invoked tensor operations create short kernels and CPU-mediated transfers interrupt continuous execution.
  • 64% of each rollout step is post-physics work versus 32% physics, while physics simulation itself accounts for only 22.3% of total rollout time.Outside physics simulation, the GPU is active for only 7.3% of total wall time on average.
  • The Isaac Lab body_com_pose_b accessor copies CPU-resident simulation data to the GPU on every call, aligning with an approximately 15.9% GPU stall.
  • A single .item() call reads GPU termination data on the CPU before reset, forcing a full GPU synchronization.

4 GPU-Native Training Loop

The GPU-native training loop reduces rollout inefficiency by eliminating CPU memory access and reducing GPU kernel fragmentation through fusion and graph replay.

  • 4 GPU-Native Training Loop: The loop applies kernel fusion and CUDA Graph replay across post-physics computation and PPO updates while keeping state access and reset handling on the GPU.It is built on MuJoCo Warp, which exposes simulator state as GPU arrays.
  • 4.1 Reducing GPU Kernel Fragmentation: Partial fusion yields a 2.5× speedup (281 →112 μs), while CUDA Graph replay yields a 9.7× speedup (29 μs).These variants isolate the effects of fusion and launch-gap removal before combining both techniques.
  • 4.1 Reducing GPU Kernel Fragmentation: 89.7× speedup reduces reward computation to 3.14 μs by combining kernel fusion with elimination of launch gaps.The warp variant uses one GPU kernel and one launch, approaching fragmentation-free execution.
  • 4 GPU-Native Training Loop: CPU memory access stalls the GPU through simulator-state copies and synchronization when reset logic reads termination flags with .item().The state accessor copies CPU-resident data to the GPU, while reset transfers a scalar back to the CPU.

5 LLM Agent Based Hyperparameter Optimization

InstantMimic supports rapid LLM-agent-driven hyperparameter optimization by reducing controller evaluation and training costs to seconds. The agent iteratively interprets metrics while maintaining search state, targeting train time to reach over 0.9 success on backflip tracking.

  • 5 LLM Agent Based Hyperparameter Optimization: Seconds-scale controller evaluation reduces hyperparameter-search cost compared with conventional trials requiring substantial training time.The framework enables alternatives to manual tuning or coarse search strategies.
  • 5 LLM Agent Based Hyperparameter Optimization: The LLM agent iteratively interprets training metrics and reshapes the hyperparameter search space as runs progress.Its persistent state includes a natural-language strategy, prior candidates and results, incumbent parameters, and globally best parameters.
  • 5 LLM Agent Based Hyperparameter Optimization: The backflip tracking objective is the train time required to exceed a 0.9 success rate.
  • 5 LLM Agent Based Hyperparameter Optimization: Three representative optimization plans correspond to regions marked (i)–(iii) in Figure 7a.

6 Results

InstantMimic improves end-to-end training throughput and enables seconds-level learning across diverse motion-tracking tasks. Its LLM-driven search further accelerates backflip training, while latent-controller pretraining supports downstream goal tracking.

  • 6.1 Training Throughput: 0.613 Mfps is 5.85× Isaac Lab and 3.91× mjlab on backflip tracking in the complete training loop.The full-loop speedup is lower than the 89.7× reward-computation speedup because simulation, inference, and PPO updates remain included.
  • 6.2 LLM Hyperparameter Optimization: 2.17 s versus 18.77 s reduces backflip time to success by 8.6× after 80 LLM-agent search cycles.Average pose error is also reported as a secondary tracking-quality metric.
  • 6.2 Motion Tracking: 1.5–4.5 s is sufficient for all five reference motions, spanning walk, run, spinkick, backflip, and cartwheel.Learning curves and qualitative tracking results cover the five motions; cartwheel is the most challenging and takes about 4.5 seconds.
  • 6.3 Analysis of the LLM Search: The LLM agent modifies search structure by diagnosing sub-metric regressions, switching hyperparameter classes, and expanding boundary-limited search ranges.These decisions are tied to cycle-log evidence and are contrasted with fixed-search-space optimization.
  • 6.4 Scaling to Large Motion Datasets: 30 minutes of AMASS pretraining reaches PHC-comparable success, and one minute of downstream training reaches 8.2 goals per 10-second episode.The latent controller is pretrained on 37.4 hours of AMASS motion before downstream goal-position tracking.

7 Conclusion

The system attributes training inefficiency to surrounding infrastructure rather than simulation alone and addresses it with GPU-native execution. It reaches seconds-level motion tracking and makes automated experimentation practical, while HPO remains sensitive to its ranking objective.

  • 7 Conclusion: Kernel fragmentation around physics and CPU-mediated memory access underutilize the GPU, while optimized kernels and direct GPU-memory access improve throughput.The conclusion identifies these rollout-path inefficiencies as the primary source of baseline underutilization.
  • 7 Conclusion: Seconds-level training covers standard motion tracking, while large-scale 37.4-hour AMASS latent-controller pretraining takes 30 minutes.These are reported as resulting system training times.
  • 7 Conclusion: 18.77 s to 2.17 s over 80 cycles shows that LLM-agent optimization can iteratively refine training configurations for backflip.The reported speedup is presented as a consequence of making automated experimentation practical.
  • 7 Conclusion: HPO favors faster convergence over tracking quality when tracking metrics are diagnostic rather than part of candidate ranking.At cycle 21, a candidate improved time to success but exhibited higher tracking errors.
  • 7 Conclusion: The findings suggest that reducing iteration time makes automated search increasingly powerful for controller design.This is stated as the paper’s broader implication for modern GPU RL pipelines.

A.1 Agent Interface and Search Protocol

The agent interface separates planning, proposal generation, validation, and experimentation while preserving search state across cycles. Prompts expose task, search-space, and current-state information so the agent can revise strategy and propose executable candidates.

  • A.1 Agent Interface and Search Protocol: The executor invokes an LLM for plan and propose stages, then handles state updates and experiments.The authored prompts instruct the model not to access files or invoke tools.
  • A.1 Agent Interface and Search Protocol: Persistent search state includes strategy, prior candidates and results, incumbent parameters, and globally best parameters.These artifacts are carried across optimization cycles.
  • A.1 Agent Interface and Search Protocol: The plan prompt revises strategy and selects the next hyperparameter axis, while the propose prompt returns candidate values as JSON.Both prompts combine accumulated evidence with the resolved task configuration.
  • A.1 Agent Interface and Search Protocol: Accepted proposals are trained and benchmarked, whereas validation failures are fed into the next propose prompt.The executor validates every proposal against the search space before training.

A.2 Backflip Search

The backflip search minimizes median active training time to a defined success threshold across three seeds and reports diagnostic tracking metrics separately. Across repeated searches, GPT-5.5-high is more consistent than GLM-5.2-high, although GLM finds the fastest single run.

  • A.2 Backflip Search: Global mean per-joint position error and end-effector error are reported as diagnostic metrics alongside the time-based objective.These metrics do not define the primary ranking criterion.
  • A.2 Backflip Search: 2.17 s at cycle 65 reduces backflip time to success from 18.77 s under the defined 80-cycle search.The objective minimizes the median of three seed-level active training times to reach success rate 0.9.
  • A.2 Backflip Search: GPT-5.5-high achieves a 2.291 s median best time with 0.095 s IQR across five independent searches.All five GPT-5.5-high results fall between 2.031 and 2.449 seconds.
  • A.2 Backflip Search: GLM-5.2-high has a 2.530 s median and 0.571 s IQR, but one search reaches 1.748 s, the fastest result across all ten searches.The comparison uses five searches per LLM under the same configuration.
Loading 2609.09821v1…