PaperlayerView exact revision on arXiv

Source-linked AI summary

A Unified Framework for Empowerment and Predictive Control

Wooyoung Chung, Tristan Shah, Volodymyr Makarenko, Stas Tiomkin

arXiv:2610.03563v1cs.RO

TL;DR

Sampling-based MPC can depend on informative, hand-designed costs, while standard empowerment separates virtual probing from executed control. This paper unifies probing and acting in an MPC formulation using first-order dynamics sensitivities; its empirical results report improved closed-loop performance and stabilization under sparse feedback. The conclusion also notes sensitivity to planning-horizon choices.

  • Problem

    Sampling-based MPC depends on informative costs that can require substantial domain knowledge, while standard empowerment separates probing actions from executed control.

    Source: §I · p. 1, §III · p. 3

  • Method

    The framework makes empowerment a trajectory-level MPC objective using a shared action distribution for probing and control, and combines it with task-specific costs.

    Source: §I · pp. 1–2

  • Results

    The authors report improved closed-loop performance and stabilization under sparse feedback, including emergent behaviors in underactuated and high-dimensional systems.

    Source: §VI · p. 5, §VII · p. 6

  • Takeaways & Limitations

    The formulation provides a way to evaluate task-agnostic empowerment and task-specific costs coherently over a shared predictive horizon.

    Source: §VI · p. 5

    MPC-based empowerment is highly sensitive to hyperparameters, particularly planning horizon, whose longer values do not monotonically improve performance.

    Source: §VII · p. 6

Abstract

from arXiv · show

Sampling-based model predictive control (MPC) is a powerful approach to trajectory optimization, but its performance depends on an informative cost function that often requires substantial domain knowledge to design. An alternative is to derive objectives directly from the system dynamics. Empowerment, defined as the channel capacity between an agent's inputs and its future state, provides such a goal-agnostic objective and has been shown to produce useful behaviors across a range of domains. However, existing empowerment-based controllers have two key limitations: empowerment is estimated using a virtual probing policy distinct from the executed control policy, obscuring its connection to the resulting behavior; and its computation typically requires second-order derivatives of the dynamics, limiting compatibility with standard MPC methods. We formulate empowerment using a single policy for both probing and acting, directly linking the empowerment objective to the executed behavior. The resulting objective requires only first-order derivatives and can be optimized with standard MPC solvers, either alone or in combination with a task-specific cost. We evaluate the proposed approach on classical control tasks and show that combining empowerment with task costs matches or exceeds the success rate achieved by either objective alone. Our formulation provides a practical bridge between standard MPC and intrinsically motivated control.

I. INTRODUCTION

Sampling-based MPC offers flexible trajectory optimization, but depends on informative, often hand-tuned costs. The paper recasts empowerment as an MPC objective, unifying probing and control while avoiding the second-order dynamics derivatives used by standard empowerment controllers.

I. INTRODUCTION

  • Sampling-based MPC optimizes simulated action trajectories, but task costs may be sparse or uninformative and auxiliary cost design can require substantial domain knowledge.MPC predicts over a finite horizon, executes only the first action, and replans.Source: §I · p. 1
  • The introductory figure illustrates goal-agnostic recovery: the hopper stabilizes after falling, while standard empowerment values oscillate and fail to stabilize it.The proposed controller's empowerment values remain stable in the figure.Source: Figure 1 · p. 1

II. EMPOWERMENT

  • Empowerment measures the channel capacity between actions and future states, offering a goal-agnostic objective, but standard methods separate virtual probing from executed control.That separation can increase sampling requirements and obscure how the objective relates to resulting behavior.Source: §I · p. 1, §II · p. 2

I. INTRODUCTION

  • The proposed trajectory-level empowerment formulation uses one action distribution for probing and control, connecting empowerment estimation to the behavior produced by executed actions.The same sampled trajectories support empowerment estimation and control optimization.Source: §I · pp. 1–2
  • Optimizing empowerment directly within sampling-based MPC requires only first-order dynamics sensitivities, avoiding the second-order derivatives standard controllers use to derive actions.The paper also proposes combining empowerment with task-specific costs along the same trajectories; the introduction reports improved performance relative to either objective alone.Source: §I · p. 2

II. EMPOWERMENT

  • Classical empowerment interprets channel capacity as the maximal achievable volume of future states under an optimal probing distribution.The paper describes a three-step derivation for continuous-domain empowerment controllers.Source: §II · p. 2

A. Dynamics as a Communication Channel

The section models the horizon-end state as a linear Gaussian communication channel by linearizing iterated dynamics around zero actions. A Gaussian probing policy and added state noise make empowerment estimation tractable.

A. Dynamics as a Communication Channel

  • The horizon-end state is expressed through iterated dynamics, then linearized around a nominal sequence of zero actions to form a communication channel.The channel approximation includes the nominal final state and the dynamics' sensitivity to the open-loop action sequence.Source: A. Dynamics as a Communication Channel · p. 2
  • A small amount of noise is added to the final state to ensure finite channel capacity.The passage defines the noise as having a Gaussian distribution.Source: A. Dynamics as a Communication Channel · p. 2
  • The probing policy is assumed to be Gaussian and centered on a zero-action mean sequence.Its covariance describes the open-loop control sequence.Source: A. Dynamics as a Communication Channel · p. 2

B. Channel Capacity via Water-filling

The water-filling calculation evaluates empowerment under an optimal probing covariance. In the classical controller, the mean action sequence remains fixed at zero; the proposed method removes that restriction so the mean can affect channel capacity.

B. Channel Capacity via Water-filling

  • Applying the linear Gaussian channel under a Gaussian probing policy enables empowerment calculation by water-filling, which optimizes the probing covariance.The resulting channel-capacity value is computed under the optimal probing covariance matrix.Source: B. Channel Capacity via Water-filling · pp. 2–3
  • The classical controller estimates empowerment, derives an action in the empowerment-gradient direction, and applies it to the dynamics.The described action rule is identified as a second-order controller.Source: B. Channel Capacity via Water-filling · p. 3
  • Unlike classical water-filling, the proposed method allows the mean action sequence to affect channel capacity instead of fixing it at zero.Classical water-filling optimizes only the probing covariance.Source: B. Channel Capacity via Water-filling · p. 3

C. Controller via The Empowerment Principle

The empowerment principle favors states with high empowerment, with a gradient-ascent controller mapping state-space empowerment gradients into control actions. The classical controller applies the selected action and repeats its channel, water-filling, and action steps, but its virtual probes are disconnected from the actions actually executed.

C. Controller via The Empowerment Principle

III. MODEL PREDICTIVE CONTROL

Sampling-based predictive control optimizes finite-horizon action sequences by evaluating candidate rollouts, making it suitable for challenging dynamics and parallel simulation. Existing empowerment methods use virtual probes and separate control actions, creating a mismatch with MPC’s single-distribution framework.

III. MODEL PREDICTIVE CONTROL

  • Sampling-based predictive control optimizes finite-horizon action sequences by evaluating candidate rollouts under a forward model and updating an action distribution to maximize predicted cumulative return.Source: §III · p. 3
  • These methods are effective for nonlinear, discontinuous, or contact-rich systems, and parallel simulation makes them suitable for real-time MPC.Source: §III · p. 3
  • Existing empowerment methods separate virtual probing actions from control actions, creating incompatible action representations and added sampling complexity for MPC objectives using one distribution.Source: §III · p. 3

IV. PROPOSED METHOD

Classical empowerment linearizes dynamics around a zero-control sequence and probes locally around its passive trajectory, but this does not naturally support trajectory planning or task objectives. The proposed method allows a nonzero-mean probing policy and optimizes that mean control sequence to guide the agent toward higher empowerment.

IV. PROPOSED METHOD

  • Classical empowerment linearizes dynamics around zero control and probes locally around a passive nominal trajectory, an assumption deemed reasonable for sufficiently small controls.Source: §IV · p. 3
  • The probing policy is Gaussian around a possibly nonzero mean control sequence, which can be optimized to guide the agent toward higher-empowerment states.Source: §IV · p. 3
  • The mean control sequence is iteratively optimized and its first action is applied; the final-state covariance depends on that mean sequence.Source: §IV · p. 3

A. Novel Formulation for Single-Policy Empowerment

The formulation generalizes empowerment through nested optimization over the action-distribution mean and covariance, reducing to standard empowerment when the mean is zero. It uses water-filling for the covariance optimization and zero-order MPC trajectory optimization with only first-order dynamics sensitivities for empowerment evaluation.

A. Novel Formulation for Single-Policy Empowerment

B. Algorithm

The controller evaluates empowerment and any task cost on sampled perturbations of a nominal control sequence, then applies the optimized sequence in receding-horizon fashion. The weighting parameter selects the relative role of task and intrinsic objectives.

B. Algorithm

  • At each timestep, the controller evaluates empowerment and task cost on N perturbed control sequences around a nominal sequence.The nominal sequence is refined through online interaction; the combined objective is evaluated for each sample.Source: B. Algorithm · p. 4
  • The controller applies the optimized sequence’s first input, then shifts the sequence forward to warm-start optimization at the next state.This repeats until task completion.Source: B. Algorithm · p. 4
  • The parameter β controls the relative importance of the task objective, while β = 1 with the task objective removed gives a purely intrinsic controller.The intrinsic objective is empowerment.Source: B. Algorithm · p. 4

V. EXPERIMENTS

The experiments examine intrinsic recovery in Hopper and task performance on pendulum systems. Empowerment combined with sparse task costs improves success, while empowerment alone can recover and stabilize the Hopper without a task-specific cost.

A. Intrinsic Self-Recovery

  • After a fall, MPC-empowerment guides the Hopper to a controllable crouch, where it maintains stability indefinitely without a task-specific cost.The crouched position is reported as more controllable than the fully upright position.Source: A. Intrinsic Self-Recovery · p. 4, §V · p. 4
  • On Hopper, standard empowerment is noisy and falls to a low value, while MPC-based empowerment remains smooth and stable.The comparison is reported despite standard empowerment having access to second-order dynamics gradients.Source: A. Intrinsic Self-Recovery · p. 4

B. Improved Uptake of Task Objective

  • The sparse-cost swing-up tasks begin with pendulums downward and provide informative feedback only after reaching the upright-position cone.The double-pendulum objective requires both links to be upright simultaneously.Source: B. Improved Uptake of Task Objective · p. 5
  • Combining empowerment with sparse task costs improves task success across all sampling algorithms, whereas sparse task cost alone fails to swing up and stabilize the tested pendulum systems.The evaluation includes single pendulum, cart-pole, and double pendulum tasks.Source: §V · p. 4, B. Improved Uptake of Task Objective · p. 5

VI. DISCUSSION

The discussion emphasizes a unified probing-and-control distribution, first-order empowerment evaluation, and improved performance when empowerment accompanies sparse task costs. These choices support a trajectory-level empowerment objective within MPC.

VI. DISCUSSION

  • The unified framework evaluates task-agnostic capacity metrics and task-specific costs coherently over a shared horizon, with empirical gains in sparse-feedback stabilization.The discussion also reports emergent behaviors in underactuated and high-dimensional systems.Source: §VI · p. 5

A. Unification of Probing and Control Distributions

  • The probing distribution can have a nonzero mean that MPC directly optimizes, aligning capacity-estimation samples with executed control inputs.This reframes empowerment as an online, trajectory-level objective compatible with sampling-based MPC.Source: A. Unification of Probing and Control… · p. 5

B. First Order Control

  • Control uses zero-order trajectory optimization and empowerment evaluation uses first-order linearizations, avoiding the second-order dynamics derivatives required by prior controllers.The discussion identifies real-time use in complex robotic systems as a relevant application scope.Source: B. First Order Control · p. 5

C. Improved Uptake of Task Objective

  • Across single pendulum, cart-pole, and double pendulum, combined empowerment and sparse task cost outperform empowerment alone; sparse task cost alone fails to solve the tasks.The table reports success rates after the initial swing-up transient, with systems initialized downward.Source: C. Improved Uptake of Task Objective · pp. 5–6, Table I · p. 6

VII. CONCLUSION

Empowerment within MPC can serve as a task-agnostic objective for complex behaviors and faster convergence under sparse performance costs, but its use remains constrained by horizon sensitivity and rollout demands.

VII. CONCLUSION

  • Empowerment-guided MPC supports complex behaviors such as self-righting and faster convergence under sparse performance costs, including without dense task supervision.The conclusion describes self-righting in underactuated, high-dimensional systems.Source: §VII · p. 6
  • MPC-based empowerment is highly sensitive to hyperparameters, particularly planning horizon, and longer horizons do not monotonically improve performance.Choosing an appropriate horizon remains an open challenge in empowerment-based control more broadly, [3],.Source: §VII · p. 6
  • Stable MPC-based empowerment optimization requires many parallel rollouts, particularly in complex environments.Zero-order MPC with well-engineered task objectives can often solve challenging tasks with comparatively fewer samples.Source: §VII · p. 6
  • Scaling empowerment to higher-dimensional systems calls for more robust estimators beyond linear-Gaussian assumptions or alternative intrinsic-motivation formulations.The conclusion identifies diffusion-based estimators and Kolmogorov–Sinai entropy as possible directions.Source: §VII · p. 6
Loading 2610.03563v1…