PaperlayerView exact revision on arXiv

Source-linked AI summary

ForgetMimic: Motion Unlearning for Reinforcement Learning Humanoid Control

Xukun Luan, Zhongxiang Lei, Chen Gong, Shaowei Li, Yuanguo Bi, Jinyan Liu

arXiv:2609.28378v1cs.ROcs.CRcs.LG

TL;DR

Humanoid control policies can learn diverse motions, but selectively removing specific motions remains underexplored despite copyright and privacy concerns. ForgetMimic applies anti-reward fine-tuning for target motions and addresses training mechanisms that conflict with unlearning. Across 12 motions and two robot embodiments, it reports forgetting designated motions while preserving non-target performance.

  • Problem

    Removing specific motions from humanoid policies for copyright or privacy reasons remains an underexplored unlearning problem.

    Source: §1 · p. 2

  • Method

    ForgetMimic uses anti-reward fine-tuning to penalize tracking on target motions while retaining the original reward for non-target motions, and disables RSI and AW interference.

    Source: §4.1 · p. 4, §4 · p. 4, §4.2.2 · p. 6

  • Results

    Across two robot embodiments and 12 motions, target success rates fell from 100% to 0%, while non-target success rates remained above 85%.

    Source: §1 · p. 2, §5.2.1 · p. 7

  • Takeaways & Limitations

    The authors present ForgetMimic as a technical reference for unlearning in humanoid control.

    Source: §6 · p. 9

Abstract

from arXiv · show

Humanoid control, leveraging human demonstrations, has achieved diverse, agile, and natural locomotion behaviors through reinforcement learning (RL). While this paradigm has yielded remarkable performance in physical humanoid control, how to eliminate specific motions from learned policies remains insufficiently explored. Addressing this issue is motivated by pressing safety and privacy concerns: the removal of malicious, poisoned, or suboptimal motions, as well as copyright-protected motions subject to the right to be forgotten under regulations such as the GDPR, is of critical importance. To this end, we propose {ForgetMimic}, the first motion-level unlearning method designed specifically for physical-world humanoid control. The core idea of ForgetMimic is as follows: given a policy $\pi_\theta$ trained on $N$ motions, our method degrades performance on a target subset of $K$ motions while preserving the effectiveness of the remaining $N-K$ motions. Furthermore, we identify and resolve two key training mechanisms in robot control that lead to unlearning failure. We conduct extensive experiments on the Unitree G1 and H2 humanoid robots across 12 motions, including Dance, Fight, Flip, and others. Experimental results demonstrate that ForgetMimic effectively eliminates memory of designated motions while maintaining the normal operation of all other motions.

1 INTRODUCTION

Humanoid policies learn broad, human-like motion repertoires, but selectively removing motions is motivated by copyright and privacy concerns. ForgetMimic targets that gap and identifies training mechanisms that can resist unlearning.

1 INTRODUCTION

  • RL coordinates many robot joints and actuators to produce agile, human-like behaviors learned from demonstrations.Figure 1 depicts the goal of forgetting Fight while preserving other motions in simulation and the real world.Source: §1 · p. 1, Figure 1 · p. 2
  • Copyright and privacy concerns motivate removing specific motions from learned policies.The paper names ForgetMimic as its investigation of motion unlearning for humanoid control.Source: §1 · p. 2
  • Prior RL unlearning work largely concerns theory or agent-based games, while humanoid control adds mechanisms such as RSI and AW that compensate for underperforming actions.Sim-to-real transfer and the requirement for natural, aesthetic motions also constrain existing unlearning methods.Source: §1 · p. 2
  • ForgetMimic is proposed as a motion-unlearning method for physical-world humanoid control, evaluated across two robot embodiments and 12 distinct motions.It uses anti-reward fine-tuning while aiming to preserve the performance of motions not targeted for removal.Source: §1 · p. 2

2 RELATED WORK

Prior unlearning research addresses supervised learning, RL environments, offline trajectories, or finite tabular agents, but not physical humanoid control. The paper frames motion-level unlearning as a better fit for humanoid policies trained across motions in one environment.

2 RELATED WORK

  • Humanoid control research has developed generalist policies that execute diverse whole-body skills from large-scale and heterogeneous motion data.Shared policies can accumulate outdated, erroneous, or unsafe motions; removal may also be required for privacy, copyright, or changing needs.Source: §2 · p. 3
  • Existing RL unlearning work includes environment-level unlearning online and trajectory-level frameworks or benchmarks offline, whereas humanoid control fits multiple motions within one environment.The paper distinguishes this setup from offline RL trajectories consisting of actions, rewards, and states.Source: §2 · p. 3
  • Humanoid-specific RSI and AW compensate for underperforming motions, while exact-unlearning theory for finite discrete tabular agents does not directly fit robots’ continuous state and action spaces.The paper introduces motion-level unlearning as better suited to humanoid control.Source: §2 · p. 3

3 PRELIMINARIES

The preliminaries define a humanoid policy trained to track a library of motion clips and state the selective-unlearning objective. They also describe the tracking reward as a weighted sum of exponential-kernel terms.

3 PRELIMINARIES

  • A PPO-pretrained stochastic policy tracks reference clips from a motion library, receiving observations that encode tracking error and sending actions to the joint controller.The robot has d actuated joints, and the policy is written πθ(at|ot).Source: §3 · p. 3
  • Selective motion unlearning degrades policy performance on K target clips while minimizing impact on the remaining clips.The constraint bounds tolerable degradation on kept motions by ϵ relative to their original performance.Source: §3 · pp. 3–4
  • The tracking reward sums M exponential-kernel terms, each measuring fidelity for a specific tracking aspect.Each term uses squared tracking error, a scale parameter, kernel-sharpness parameter, and weight; regularizers include action smoothness, joint limits, and a survival bonus. The total training reward is maximized.Source: §3 · pp. 3–4

4 METHOD

ForgetMimic uses anti-reward fine-tuning to degrade target-motion performance while preserving other motions, and disables training mechanisms that otherwise counteract unlearning.

4 METHOD

  • ForgetMimic combines anti-reward fine-tuning with overrides to mechanisms that conflict with unlearning, aiming to degrade target motions while preserving other motions.Source: §4 · p. 4

4.1 ANTI-REWARD FINE-TUNING

  • For target motions, anti-reward fine-tuning adds a penalty proportional to tracking quality; non-target motions retain the original training reward.Source: §4.1 · p. 4

4.1.1 TOTAL REWARD UNDER UNLEARNING

  • The unlearning reward applies anti-reward only to target motions, leaving the original reward unchanged for non-target motions.The method is designed to selectively degrade specific motions.Source: §4.1.1 · pp. 4–5

4.1.2 GRADIENT INTUITION

  • For non-target motions, the unchanged reward preserves the original gradient direction; weakening target-motion reward reduces the gradient's tendency to reproduce them.Source: §4.1.2 · p. 5

4.2 CRITICAL MECHANISM OVERRIDES

  • Reference State Initialization (RSI) and Assistive Wrench (AW) encourage poorly performing motions, directly conflicting with unlearning.Source: §4.2 · p. 5

4.2.1 REFERENCE STATE INITIALIZATION

  • As anti-reward raises target-motion failure levels, RSI increases their sampling probability, creating a feedback loop that gives target motions more practice; RSI is disabled during unlearning.Disabling RSI ensures all motions are sampled with equal probability regardless of tracking performance.Source: §4.2.1 · p. 6

4.2.2 ASSISTIVE WRENCH

  • As target-motion failure rises, AW provides more physical assistance, counteracting the anti-reward penalty; physical assistance is disabled during unlearning.The assistive wrench applies corrective forces and torques to the torso when the policy struggles with a motion.Source: §4.2.2 · p. 6

5 EXPERIMENT

ForgetMimic selectively degrades designated motions while preserving non-target performance across a 12-motion humanoid-control policy. Experiments examine the setup, forgetting effectiveness, training mechanisms, stability over training, and generalization across motion types.

5.1 EXPERIMENTAL SETUP

  • Experiments use a 12-motion human-demonstration library on Unitree G1 and H2 robots, with Fight and FightAndSports1 as default targets.Evaluation covers 20 episodes per motion and reports tracking reward, success rate, episode length, and degradation ratio.Source: §5.1 · pp. 6–7, Figure 2 · p. 7

5.2.1 UNLEARNING VALIDITY ANALYSIS

  • Target success rates fall from 100% to 0%, while non-target tracking rewards decrease by less than 2% and success rates remain above 85%.The degradation ratio falls from 0.997 to 0.851 (λ = 8.0) or 0.646 (λ = 16.0); the higher penalty causes slight reductions on some non-target motions.Source: §5.2.1 · p. 7, Table 1 · p. 8

5.2.2 ABLATION AND COMPARISON

  • With both RSI and AW disabled, unlearning is faster; either mechanism alone shows resistance, and ARFT with AW substantially harms non-target motions.That resistance weakens with more fine-tuning epochs, which requires additional computation.Source: §5.2.2 · p. 8, Table 1 · p. 8
  • Ours (λ = 8) and Ours (λ = 16) preserve non-target performance better than the compared unlearning baselines while reducing target success rates to 0%.Ye et al. (2025) reaches a lower target tracking reward (0.5027) than Ours (λ = 16, 0.5972) and Ours (λ = 8, 0.6653), but its non-target performance collapses.Source: §5.2.2 · p. 8, Table 1 · p. 8
  • At λ = 8.0, target success reaches 0% and the degradation ratio is 0.756, while non-target tracking rewards remain above 0.85.At λ = 16.0, the degradation ratio falls further to 0.628 and target tracking reward to 0.54; the passage recommends λ greater than tracking weight W for fast, effective unlearning.Source: §5.2.2 · pp. 8–9, Table 2 · p. 9

5.2.3 CATASTROPHIC UNLEARNING ANALYSIS

  • Across 99,999 fine-tuning iterations, target rewards degrade while non-target mean rewards remain stable at 0.84–0.88, with final mean 0.868 versus baseline 0.855.The target-motion heatmap rewards shift from 0.86 before unlearning to 0.52–0.61 during unlearning.Source: §5.2.3 · p. 9, Figure 3 · p. 9

5.2.4 GENERALIZATION ACROSS MOTION TYPES ANALYSIS

  • The same unlearning procedure achieves effective forgetting when Fight, Dance, or Run motions are selected as targets.The reported target sets vary across those motion categories, with remaining motions used to assess preservation.Source: §5.2.4 · p. 9

6 CONCLUSION

The conclusion presents ForgetMimic as a motion-level unlearning method for RL-driven humanoid control. It highlights the importance of removing unwanted motions and points to results across different target motion types.

6 CONCLUSION

  • ForgetMimic is presented as the first motion-level unlearning method for RL-driven humanoid control.The conclusion frames removal of outdated, inferior, poisoned, and privacy-restricted motions as important for humanoid control training.Source: §6 · p. 9
  • Figure 4 compares Fight/FightAndSports1, Dance, and Run as target motions, with highlighted regions marking decreases in their tracking rewards.Source: Figure 4 · p. 10
  • The conclusion calls for safer, more controllable, and more privacy-compliant humanoid control.Source: §6 · p. 10

A.1 UNITREE G1 VISUALIZATIONS

The appendix presents simulation experiments on mjlab and real-world experiments on the Unitree G1, and recommends controlled, safe testing to avoid unnecessary robot damage.

A.1 UNITREE G1 VISUALIZATIONS

  • The appendix reports mjlab simulation and Unitree G1 real-world experiments, recommending controlled and safe testing to avoid unnecessary robot damage.Source: §A.1 · p. 13

A.2 FINE-GRAINED BEHAVIOR QUANTIFICATION ANALYSIS

Fine-grained physical metrics show that unlearning changes target-motion execution, not just tracking reward. The two unlearning strengths have similar trends overall, with λ = 16 showing stronger suppression on some measures.

A.2 FINE-GRAINED BEHAVIOR QUANTIFICATION ANALYSIS

  • Unlearned target-motion episodes fall from 500 steps to about 8–13, and foot contact ratio falls from 0.94 to about 0.40–0.56.These measures complement tracking reward by assessing physical behavior during target-motion execution.Source: §A.2 · p. 13, Figure 6 · p. 14
  • Mean anchor position error decreases from 0.576 to about 0.053–0.055, while joint limit violation rate rises from 0.12 to about 0.26 after unlearning.The passage associates the lower position error with the policy no longer actively following the reference before termination, and the higher violation rate with physically abnormal configurations under target motions.Source: §A.2 · p. 13
  • Compared with λ = 8, λ = 16 produces slightly shorter Fight episodes (7.8 vs. 11.0 steps) and a lower foot contact ratio (0.38 vs. 0.57), suggesting stronger behavioral suppression.Source: §A.2 · p. 14

A.3 TRAINING DYNAMICS ANALYSIS

Training dynamics distinguish successful forgetting from policy collapse: Config B forgets the target motion, while Config A retains substantial target-motion memory. The reported metrics connect this contrast to reward, entropy, wrench assistance, and sampling behavior.

A.3 TRAINING DYNAMICS ANALYSIS

  • Config B forgets the target motion as its anti-reward rises from −1.11 to −0.05 and target tracking reward falls from 0.14 to 0.006; Config A retains substantial memory.Config B's policy entropy rises from 22.0 to 31.6, while Config A's entropy falls from 12.9 to 11.9.Source: §A.3 · pp. 14–15
  • Figure 8 treats more optimization iterations needed to recover target performance as greater relearning difficulty and stronger evidence of forgetting.The figure concerns Fight and FightAndSports1 after unlearning with Ours, λ = 16.Source: Figure 8 · p. 15
  • Config A's small penalty, corrective wrench assistance, and adaptive sampling coincide with failed unlearning, whereas Config B disables the wrench and uses uniform sampling.For Config A, wrench gain rises from 0.007 to 0.026 and RSI top-1 probability from 0.013 to 0.015; Config B keeps wrench gain at 0 and top-1 probability at 0.0025.Source: §A.3 · p. 15

A.4 RELEARNING-BASED FORGETTING EVALUATION

Recovery fine-tuning tests how readily an unlearned policy can reacquire forgotten motions. The reported recovery costs are substantially greater than the cost of unlearning.

A.4 RELEARNING-BASED FORGETTING EVALUATION

  • About 4,250 iterations for Fight, 5,500 for FightAndSports1, and 7,500 jointly are needed to recover 95% of base target performance.Recovery fine-tuning starts from the λ = 16 unlearned policy and uses original positive training rewards for 10,000 iterations.Source: §A.4 · p. 15
  • Recovery costs about 8× the optimization cost of unlearning, indicating that forgotten motions are substantially harder to reacquire.The paper uses recovery cost as an indirect measure of residual target-motion knowledge.Source: §A.4 · p. 15
  • The unlearned policy begins recovery with target tracking reward 0.546, episode length 3 steps, and success rate 0%, versus base tracking reward 0.8716 and success rate 95%.These initial metrics contrast the unlearned policy with the base policy before recovery fine-tuning.Source: §A.4 · p. 15

A.5 MEMBERSHIP INFERENCE ANALYSIS

The section reports membership-inference results, implementation settings, H2 experiments, and discussion of the method's scope. H2 tests suppress target motions across target-set sizes while largely preserving non-target performance, but the paper describes its unlearning as approximate rather than certified.

A.5 MEMBERSHIP INFERENCE ANALYSIS

  • Target-motion membership inference accuracy is 0.60 for Base, 0.54 for Retraining, and 0.53 for ForgetMimic; the analysis does not rule out residual information.The study uses 12 motions, fixed target-motion probes, and 1,200 rollouts, and leaves a full sample-level benchmark for future work.Source: §A.5 · pp. 15–16

A.6.1 PPO HYPERPARAMETERS

  • PPO training uses 24 steps per environment per iteration, 5 epochs, 4 mini-batches, and a 10^-3 adaptive learning rate.The passage also specifies γ = 0.99, GAE λG = 0.95, clipping ϵPPO = 0.2, and Adam optimization.Source: §A.6.1 · p. 16

A.6.2 REGULARIZATION REWARDS

  • Regularization rewards penalize action-rate changes, joint acceleration, joint-limit violations, and torque above 90% of maximum, while a +1.0 survival reward is used.Weights are −0.1, −5 × 10^-6, −1.0, and −0.1 for action rate, joint acceleration, joint limit, and torque, respectively.Source: §A.6.2 · p. 16

A.6.3 REWARDS DESIGN

  • Tracking rewards use an exponential kernel with κ = 0.25; the passage directs readers to Table 3 for tracking-term weights.Source: §A.6.3 · p. 16

A.6.4 TERMINATION CONDITIONS

  • Episodes terminate at 10 s or when specified anchor, end-effector, or keybody-contact thresholds are exceeded.The thresholds are anchor z-position error 0.35 m, anchor orientation error 0.8 rad, end-effector z-position error 0.25 m, and keybody ground-contact force 2000 N.Source: §A.6.4 · p. 16

A.6.5 SIMULATION

  • Simulation uses MuJoCo with a 0.005 s timestep and decimation factor 4, yielding a 50 Hz control frequency on flat terrain.The setup sets nconmax to 35 and njmax to 250.Source: §A.6.5 · pp. 16–17

A.7 EXPERIMENTS ON UNITREE H2

  • Across H2 target sets of 1, 2, and 3 motions, target success falls to 0%, while non-target tracking and success remain largely comparable to Base.Target tracking rewards fall from 0.825 to 0.681 for Dance, 0.794 to 0.654 for Fight&FightAndSports1, and 0.880 to 0.631 for Flips; Figures 9 and 10 show simulation and real-world results.Source: §A.7 · p. 17, Table 4 · p. 17, Figure 9 · p. 18

A.8 DISCUSSION

  • ForgetMimic provides approximate, not provably certified, unlearning; coupled motion representations make complete information removal while preserving related behaviors challenging.The formulation is not restricted to one humanoid embodiment, and future unlearning work must account for interfering mechanisms and recovery during subsequent training.Source: §A.8 · pp. 17–18
Loading 2609.28378v1…