Learning Smooth Time-Varying Linear Policies with an Action Jacobian Penalty

Zhaoming Xie; Kevin Karol; Jessica Hodgins

Source-linked AI summary

Learning Smooth Time-Varying Linear Policies with an Action Jacobian Penalty

Zhaoming Xie, Kevin Karol, Jessica Hodgins

arXiv:2602.18312v1Feb 20, 2026cs.ROcs.GR

TL;DR

Reinforcement learning policies can exploit unrealistic high-frequency signals, while existing smoothing penalties often require substantial tuning. The paper directly penalizes the action Jacobian and introduces a Linear Policy Net to reduce the resulting computational burden. The combined approach learns smooth policies across challenging imitation tasks, including dynamic motions and control of a physical quadrupedal robot with an arm.

Problem
Reinforcement learning policies can produce unrealistic high-frequency control signals, and existing action-change penalties often require substantial task-dependent tuning.
Method
The paper combines an action Jacobian penalty with a Linear Policy Net that generates a time-varying linear feedback policy from reference-motion information.
Results
The combined method learns smooth policies for motion imitation tasks including backflips and parkour, and controls a physical quadrupedal robot with an arm.
Takeaways & Limitations
Time-varying linear policies can support smooth motion imitation across simulated characters and physical quadrupedal robots.
Takeaways & Limitations
The action Jacobian penalty does not directly constrain temporal action derivatives and is therefore not guaranteed to reduce action changes during rapidly changing motions such as backflips.

Abstract

from arXiv · show

Reinforcement learning provides a framework for learning control policies that can reproduce diverse motions for simulated characters. However, such policies often exploit unnatural high-frequency signals that are unachievable by humans or physical robots, making them poor representations of real-world behaviors. Existing work addresses this issue by adding a reward term that penalizes a large change in actions over time. This term often requires substantial tuning efforts. We propose to use the action Jacobian penalty, which penalizes changes in action with respect to the changes in simulated state directly through auto differentiation. This effectively eliminates unrealistic high-frequency control signals without task specific tuning. While effective, the action Jacobian penalty introduces significant computational overhead when used with traditional fully connected neural network architectures. To mitigate this, we introduce a new architecture called a Linear Policy Net (LPN) that significantly reduces the computational burden for calculating the action Jacobian penalty during training. In addition, a LPN requires no parameter tuning, exhibits faster learning convergence compared to baseline methods, and can be more efficiently queried during inference time compared to a fully connected neural network. We demonstrate that a Linear Policy Net, combined with the action Jacobian penalty, is able to learn policies that generate smooth signals while solving a number of motion imitation tasks with different characteristics, including dynamic motions such as a backflip and various challenging parkour skills. Finally, we apply this approach to create policies for dynamic motions on a physical quadrupedal robot equipped with an arm.

1 Introduction

Deep reinforcement learning policies can produce unrealistic high-frequency control signals, and existing smoothing methods often require task-dependent tuning. The paper proposes an action Jacobian penalty and Linear Policy Net to reduce these signals while retaining performance across challenging motion tasks.

Motivation: Deep reinforcement learning policies may favor unrealistic high-frequency control signals even when imitating high-quality motion data.Sensitivity to slight input changes makes deployment more problematic with noisy sensors and limited actuator bandwidth.
Motivation: Existing action-change penalties and input randomization reduce sensitivity but often require substantial task-dependent trial and error.Lipschitz-constrained policies also incur sampling and backpropagation costs when estimating action sensitivity.
Approach: The action Jacobian penalty penalizes the control action's sensitivity to character-state changes, reducing high-frequency signals across motion imitation tasks.The method directly targets the action Jacobian rather than relying on a task-specific action-rate reward.
Approach: The Linear Policy Net reduces Jacobian-penalty computation by producing a feedback matrix from task information, which then operates on character state.Jacobian evaluation and its parameter gradient become a simple neural-network forward and backward pass.
Results: A Linear Policy Net with the Jacobian penalty matches or surpasses fully connected policies across walking, backflip, table-tennis footwork, and parkour tasks.The paper also reports control of a physical quadrupedal robot with an arm performing dynamic hopping and arm-swing motions.

2 Related Work

Prior work uses reinforcement learning, filtering, action-rate penalties, sampling, and model-based control to produce smooth or linear-feedback policies. These approaches can require manual features, task-specific design, costly computation, or fail to generalize to complex interactions.

Deep Reinforcement Learning: Deep reinforcement learning scales across physics-based animation tasks, including scene interaction and object manipulation, while motion capture supports realistic animation imitation.The related work positions DRL as both scalable and versatile for specialized behaviors.
Smooth Control Policies: Action filters can reduce high-frequency signals but significantly reduce perturbation response, while action-rate rewards remain a prevailing smoothing strategy.High-frequency control can otherwise degrade motion quality and cause sim-to-real failures in robotics.
Linear Feedback Control: Segmented linear feedback controllers require substantial human effort because each motion needs custom segmentations and input features.Sampling-based methods learn robust controllers for dynamic motions but typically require complex, motion-specific state representations.
Linear Feedback Control: Sampling-based linear controllers have not been shown to generalize to complex environment interactions such as vaulting and wall climbing.The cited limitation concerns motions requiring richer interaction with the environment.
Linear Feedback Control: Learned linear-feedback policies can either exploit unrealistic high-frequency leg motions or depend on carefully designed features.Model-based time-varying controllers are often brittle and may require computationally expensive online replanning.
Linear Feedback Control: Linearized-equation approaches enable small online quadratic programs, but inequality constraints such as friction cones leave the resulting controllers nonlinear.These methods have been applied to quadrupedal and bipedal locomotion.

3 Problem Setup and System Overview

The paper formulates motion imitation as a DeepMimic-style control problem in which a policy maps character and reference states to joint targets. It trains these policies with PPO in simulation and combines a Linear Policy Net with a Jacobian regularizer for smooth behavior and efficient training.

Problem Formulation: A reference motion supplies the desired full character state at each timestep, and the policy maps the current and reference states to joint-angle control actions.A joint-level proportional-derivative controller actuates the target angles in physics simulation.
Training Setup: Experiments use MuJoCo with simulation at 120 Hz and policy updates at 30 Hz, comparing fully connected feedforward and Linear Policy Net architectures.Actions are sampled from a fixed Gaussian distribution with diagonal variance parameter δ^2 = 0.01.
Training Setup: PPO collects 2500 samples from 50 parallel environments per iteration and runs for up to 5000 iterations.The reported training duration is around 2.5 hours on a workstation.
System Overview: The proposed Linear Policy Net with an action Jacobian penalty introduces smooth behavior with minimal tuning and compute overhead while maintaining learning efficiency.It is presented as a regularization loss added to the training setup rather than as an additional task-specific reward term.

4 Action Jacobian Penalty

The action Jacobian penalty directly regularizes policy sensitivity to character-state changes, targeting the source of high-frequency control signals. Although exact Jacobian optimization is computationally costly for fully connected networks, autodifferentiation provides a direct formulation that the proposed architecture can make practical.

Motivation: Reinforcement learning exploration noise can create high-frequency control signals that policies fit, producing jittery behavior amplified by real-world sensor and motor imperfections.Existing consecutive-action reward penalties can have a small regularization effect and require tuning.
Penalty Formulation: The method adds the action Jacobian penalty directly to the PPO loss instead of using it as a reward signal.This places policy smoothness regularization inside the optimization objective.
Penalty Formulation: The Jacobian penalty is L_Jac = ∥J∥2, the squared Frobenius norm of the action Jacobian with respect to character state, weighted by w_Jac.The experiments use w_Jac = 10 throughout.
Interpretation: Penalizing the Jacobian norm discourages policies whose small state changes cause large action changes, thereby producing smoother control signals.The Jacobian is organized from action derivatives with respect to state dimensions.
Comparison: Direct full-Jacobian optimization through autodifferentiation and backpropagation avoids the directional and sampling limitations of prior approximations.Prior approaches either sample around data points or optimize a sampled-action-dependent direction of the Jacobian.
Computational Cost: Fully connected networks make Jacobian-penalty training about 1.5 times slower per PPO iteration than using only the PPO loss.The computational overhead motivates the Linear Policy Net architecture.

5 Linear Policy Net

The Linear Policy Net uses a reference-conditioned MLP to produce a time-varying linear feedback policy, enabling efficient action-Jacobian computation and slower policy updates than action computation. Its design omits character-state dependence while retaining successful dynamic-motion learning in simulation and on a physical robot.

Architecture: The LPN maps the reference state to a feedback matrix K_t and feedforward action k_t, then applies a_t = K_t s_t + k_t + â_t.K_t ∈ R^(m×n), where n is character-state dimension and m is control-action dimension.
Architecture: For fixed reference sequences, K_t and k_t depend only on time, making the policy a time-varying linear feedback controller.
Design choices: The LPN uses minimal character coordinates as policy input, while excluding character-state dependence from the MLP-generated feedback matrices.The authors report that this suffices for dynamic motion in simulation and on a physical robot.
Inference hierarchy: The feedback matrix and feedforward term can be updated more slowly than the action, with MLP inference reaching 15 Hz for some tasks while joint control remains faster.Legged-robot frameworks may update joint targets around 50 Hz and joint-level PD control at 500 Hz or more.
Action space: Using K_t and k_t directly as the action space would increase its dimension from m to (n + 1) × m, making learning harder.
Jacobian computation: Because the action Jacobian equals K_t, its penalty and parameter gradient require only the LPN MLP's existing forward and backward passes.The additional computation is minimal because these passes are already needed for the PPO loss.

6 Evaluations

The evaluations compare regularization methods and Linear Policy Nets across motion-imitation tasks, smoothness measures, learning speed, robustness, skill transitions, and physical-robot execution. LPNs with action Jacobian penalties generally preserve smooth control while reducing computational and inference burdens.

Evaluation Tasks: The evaluation spans humanoid locomotion, dynamic gymnastics, single-sequence imitation, environment interaction, and physical quadrupedal-robot motions.Tasks include walking, running, flips, cartwheels, table-tennis footwork, breakdancing, parkour, and soccer juggling.
Comparison Methods: The comparison includes unregularized FF policies, FF policies with action Jacobian penalties, action-change rewards at multiple weights, and Lipschitz-constrained policies.Action-change penalties use tunable weights, whereas the Jacobian penalty is evaluated with both FF and LPN architectures.
Learning Performance: LPNs with Jacobian penalties achieved the fastest convergence across walking, backflip, and table-tennis footwork, with minimal per-iteration computational overhead.FF policies with Jacobian penalties took almost twice as many iterations to converge, with each iteration about 1.5 times slower.
Smoothness: Action regularization reduced rapidly changing, jittery actions, while heavy action-change penalties could prevent learning dynamic tasks such as backflips.The LPN remained competitive in smoothness, although its backflip smoothness metric was worse than some FF Jacobian-penalty and appropriately tuned reward policies.
Robustness and Sim-to-Real: The learned policies adapted to uneven terrain, supported skill transitions after distillation, and controlled a physical Spot robot through pacing with fast arm movement.Walking feedback updates could be reduced to 10 Hz, while other motions failed below 30 Hz; on Spot, matrices were updated at 15 Hz and joint targets at 30 Hz.

7 Conclusions and Discussion

The paper presents an efficient framework for smooth time-varying linear control policies across motion imitation tasks, while identifying important scope and robustness boundaries. The approach supports broad motion coverage without special feature engineering, but remains limited in task types, motion duration, and arbitrary skill transitions.

Conclusions and Discussion: The framework efficiently trains smooth time-varying linear control policies for simulated characters and physical quadrupedal robots.The learned policies are represented in a simple linear-matrix form.
Conclusions and Discussion: Time-varying linear policies exist for a wide range of motions without special feature engineering.The paper contrasts this learned formulation with model-based differential dynamic programming, whose matrices are often less robust.
Conclusions and Discussion: The action Jacobian penalty is not guaranteed to reduce temporal action changes during dynamic motions because it penalizes state derivatives rather than time derivatives.This limitation may explain its lower effectiveness on the backflip.
Conclusions and Discussion: The system focuses on relatively short motion segments rather than the longer motion sequences supported by current deep reinforcement learning systems.Scaling to more and longer motions could enable learning a policy generator from feedback matrices.
Conclusions and Discussion: Policy distillation demonstrates skill composition, but the system cannot yet transition between arbitrary skills.The authors suggest larger motion datasets and a control graph as possible routes toward more diverse transitions.
Conclusions and Discussion: The formulation is limited to DeepMimic-style motion imitation tasks and does not yet cover tasks without motion-capture data.Extending it to adversarial motion imitation is identified as future work.