PaperlayerView exact revision on arXiv

Source-linked AI summary

Active Learning of Dynamics for Data-Driven Control Using Koopman Operators

Ian Abraham, Todd D. Murphey

arXiv:1906.05194v1cs.RO

TL;DR

The paper addresses how robots can learn useful Koopman dynamics quickly while collecting task-relevant data and retaining control capability. It develops an information-theoretic active-learning controller, demonstrates improved information acquisition and single-execution quadcopter stabilization, and extends the approach to automated observables and real robots.

  • Problem

    Robotic Koopman learning requires actuated data collection, while useful learning must incorporate task information and support control during the learning process.

  • Method

    The paper combines information-theoretic active learning with Koopman operator representations and linear-quadratic control, extending the framework to automated function observables.

  • Results

    The active-learning controller increases information about the Koopman operator, enables single-execution quadcopter learning and stabilization during free-fall, and is demonstrated on real robotic systems.

  • Takeaways & Limitations

    Koopman representations let robots learn dynamics while exploiting linear control structure, combining rapid active learning with improved control authority.

Abstract

from arXiv · show

This paper presents an active learning strategy for robotic systems that takes into account task information, enables fast learning, and allows control to be readily synthesized by taking advantage of the Koopman operator representation. We first motivate the use of representing nonlinear systems as linear Koopman operator systems by illustrating the improved model-based control performance with an actuated Van der Pol system. Information-theoretic methods are then applied to the Koopman operator formulation of dynamical systems where we derive a controller for active learning of robot dynamics. The active learning controller is shown to increase the rate of information about the Koopman operator. In addition, our active learning controller can readily incorporate policies built on the Koopman dynamics, enabling the benefits of fast active learning and improved control. Results using a quadcopter illustrate single-execution active learning and stabilization capabilities during free-fall. The results for active learning are extended for automating Koopman observables and we implement our method on real robotic systems.

I. INTRODUCTION

The paper develops task-aware active learning for robotic Koopman dynamics, combining information-driven data collection with representations that support linear-quadratic control. It motivates this approach by linking data collection, nonlinear-system representation, and control synthesis.

  • Motivation: Robotic active learning requires a controller that incorporates task information, learns a dynamic representation, and supports additional policies during learning.
  • Koopman representation: Koopman operators represent nonlinear dynamics as linear, infinite-dimensional systems by evolving functions of state over time.
  • Active data collection: Data-driven Koopman approximations require actuation to collect data, making the learning process itself an active control problem.
  • Contribution: The proposed method combines information-theoretic active learning with Koopman-based control authority and supports learning in real-time settings where only one execution is possible.
  • Relation to Previous Work: The paper extends prior Koopman-control work with new control examples, automated design choices, and active learning of Koopman dynamics.

B. Approximating the Data-Driven Koopman Operator

The paper approximates Koopman dynamics in finite observable spaces and uses the resulting linear structure for control. A Van der Pol example shows improved control performance, while also motivating active data collection because random inputs require substantial state-space coverage.

  • B. Approximating the Data-Driven Koopman Operator: A finite set of observables approximates the infeasible infinite-dimensional Koopman operator, with residual error that can decrease as the observable dimension grows.
  • Controlled Koopman dynamics: The operator can incorporate predefined control inputs through separate state and input submatrices, requiring consecutive control inputs in the dataset.
  • Control authority: Koopman dynamics map nonlinear constraints into a linear modified state-space, enabling LQ controllers that can outperform controllers based on local linearization.
  • Van der Pol example: The Van der Pol Koopman model uses 5000 randomly sampled initial conditions with uniformly random inputs to estimate dynamics.
  • Van der Pol example: In the forced Van der Pol oscillator, Koopman-based LQ control produces less overall integrated trajectory error than local linearization, while a learned state-space model matches known-dynamics results.
  • Motivation for active learning: Random-input data collection can require substantial data to cover the Van der Pol state space, motivating active learning of the Koopman operator.

IV. CONTROL SYNTHESIS FOR ACTIVE LEARNING OF KOOPMAN OPERATOR DYNAMICS

The paper formulates active learning as switching between a task policy and an information-seeking controller while retaining task objectives. Mode insertion gradients yield a closed-form switching control that can improve learning subject to task bounds.

  • Control formulation: Active learning is formulated as a hybrid switching problem between a task policy and an information-maximizing controller.The controller self-excites dynamics to collect data for estimating the Koopman operator.
  • Control formulation: The objective combines an information-maximizing learning cost with the task objective solved by the existing policy.This lets stabilization and information acquisition appear in one general objective.
  • Control formulation: During training, Koopman-derived policies may be inaccurate, but the paper assumes that the model and policy converge with experience.This is presented as a model-based reinforcement-learning assumption.
  • Control formulation: Mode insertion gradients quantify how the objective changes when the task policy switches temporarily to an arbitrary control.The sensitivity is evaluated for an infinitesimally small switching duration.
  • Control formulation: Because the secondary objective is quadratic and separable in time, the switching control admits a closed-form solution.The solution is computed over the application interval and then used to improve the objective.
  • Control formulation: The resulting switching formula improves the active-learning objective while respecting bounds on arbitrary task objectives included in the formulation.The paper states this result through Proposition 2 and its corollary.

B. Information Maximization

The paper uses Fisher information to quantify information about Koopman parameters and incorporates its inverse into the control objective. Theorem 1 states that the resulting controller increases information acquisition relative to the task policy.

  • Information measure: Fisher information is used to measure how much the observed dynamics reveal about Koopman operator parameters.The Koopman dynamics are treated as a maximum-likelihood estimation problem.
  • Information measure: The trace of the Fisher information matrix defines the T-optimality measure used for active learning.The noise covariance appears in the Gaussian formulation, and the Fisher matrix is positive semi-definite.
  • Information measure: The learning cost is set to 1/(I(K) + ϵ), so minimizing the objective maximizes the T-optimality measure.ϵ is small and prevents singular solutions caused by the positive semi-definite Fisher information matrix.
  • Information measure: The information objective assumes that positive information for an approximate operator implies positive information for the true Koopman operator.The approximation is computed from the dataset collected up to the current sampling time.
  • Information maximization: Theorem 1 gives the first-order change in information produced by the active-learning controller relative to the task policy.The comparison uses the T-optimality measures obtained under the two controls.
  • Information maximization: The controller increases the rate of information acquisition, and weighting information against task cost ensures positive relative information gain.The claim is made relative to using only the existing policy.

V. SINGLE EXECUTION ACTIVE LEARNING OF FREE-FALLING QUADCOPTERS

The paper evaluates single-execution active learning by requiring a free-falling quadcopter to learn its dynamics during the first second and then stabilize using the learned model.

  • Single-execution active learning: The free-falling quadcopter must learn a dynamics model within its first second and use it to generate a stabilizing controller.The experiment targets model learning and stabilization within one execution.

A. Problem Statement

The quadcopter experiment compares information gain and stabilization across learning strategies under matched initial conditions. The proposed method balances information acquisition with timely stabilization, whereas stronger excitation can hinder stabilization.

  • Problem Statement: The task requires learning within one second of free fall and then preventing further falling with a controller based on the learned model.Success is defined by a state error below 0.01 relative to the zero-velocity target.
  • Problem Statement: The comparison uses 20 uniformly sampled initial linear and angular velocities, resetting the learned model after each trial.This prevents information from previous trials from carrying over.
  • Problem Statement: The evaluated strategies include least-squares adaptive stabilization, two-stage motor babble, active Gaussian-process learning, and the proposed method.The Koopman model is used for the first three listed strategies and the proposed method, while Gaussian processes operate in the original state space.
  • Results: The proposed method improves information over motor babble, while other methods can obtain more information by overly exciting the dynamics.Direct adaptive stabilization eventually stabilizes despite initially using an incorrect model.
  • Results: Quick stabilization requires choosing when to learn and when to stabilize, because information-seeking approaches can drive the quadcopter toward unstable regions.The Koopman formulation further enhances control authority for several compared methods.
  • Results: The figure compares Fisher-information gain, stabilization error with standard deviation, and representative trajectories across 20 falling-quadcopter trials.The caption identifies information gain in panel (a), error over time in panel (b), and time-series snapshots in panel (c).
  • Results: The active Gaussian-process method depends on data quality and a local linear approximation, leaving insufficient nonlinear information for reliable single-execution learning.The paper reports that it succeeds only at times in this setting.

D. Sensitivity to Initialization and Parameters

The paper examines sensitivity to Koopman initialization and algorithm parameters, while extending active learning to automatically discover function observables. These choices affect exploration, control behavior, computational cost, and task performance.

  • Sensitivity to Initialization and Parameters: Reasonable non-zero Koopman initializations within an order of magnitude produce performance comparable to the main active-learning experiment.Results may vary with sampling frequency and the underlying system; a precomputed-operator quadcopter benchmark is also provided.
  • Sensitivity to Initialization and Parameters: Large regularization prevents substantial state exploration, whereas overly small regularization can widen exploration harmfully when states are unbounded.Adding weight to the active-learning objective produces a similar effect.
  • Sensitivity to Initialization and Parameters: Shorter time horizons produce more reactive behavior, while longer horizons tend to produce more intent-driven control responses.Choosing the horizon is problem specific, although the algorithm has relatively few tunable parameters beyond the task objective.
  • Sensitivity to Initialization and Parameters: Prior knowledge helps select nonlinear state functions, but random polynomial or Fourier observables can produce unstable Koopman eigenvalues that complicate model-based control.Underlying structures are often unknown or difficult to find in robotics, motivating automated observable discovery.
  • Automatic Function Discovery: Deep neural networks parameterize Koopman observables, while active learning updates the observables, Koopman dynamics, and LQ controller from collected task data.The method is evaluated on cart-pendulum stabilization and 2-link robot control.
  • Automatic Function Discovery: 50 iterations stabilize the cart pendulum versus over 100 with additive noise, while the 2-link robot reaches its target within 5 active-learning iterations.These results accompany automated learning of both the function observables and Koopman operator.

C. Discussion

The paper evaluates active Koopman learning on real robotic systems, including Sphero SPRK trajectory tracking in sand and Sawyer arm control. Active learning improves SPRK responsiveness and phase lag, while Koopman control performs comparably to Sawyer’s embedded controller.

  • Discussion: A neural-network observable-design method still has a trivial zθ, vθ = 0 solution that depends on parameter initialization.Adding state x to the neural-network output of zθ was sufficient in the authors’ experiments to overcome it.
  • Robot experiments: The experiments use Sphero SPRK in sand and a 7-link Sawyer arm to test Koopman control and active learning under challenging dynamics.The SPRK challenge is maneuvering through sand; Sawyer’s challenge is high dimensionality and inertial effects.
  • Sphero SPRK: 20 seconds of active learning precedes SPRK trajectory tracking, after which the information-maximizing weight is set to zero.The evaluation reports x−y trajectory RMSE, Pearson’s correlation, and phase lag.
  • Sphero SPRK: Active learning improves the SPRK Koopman controller’s overall responsiveness and phase lag in sand compared with the authors’ previous method.The passage explicitly states that active learning improves model-based controller performance relative to.
  • Sawyer: The data-driven Koopman controller performs comparably to Sawyer’s built-in controller for trajectory tracking.The comparison uses tracking-position RMSE, Pearson’s correlation, and phase lag; both controllers run at 100 Hz.
  • Sawyer: Online updates allow the Sawyer Koopman controller to capture individual-joint dynamic effects without a precomputed model or linearization.The paper connects continuous data acquisition with high-accuracy control and evolution of nonlinearities.

APPENDIX A PARAMETERS FOR VARIOUS EXAMPLES

The appendix specifies model, observable, control, and simulation parameters for the Van der Pol and quadcopter examples. These settings include regression-based Koopman matrices, selected velocity-product basis functions, sampling rates, and stabilization horizons.

  • Van der Pol example: The actuated Van der Pol example uses Koopman functions and linear regression to generate matrices A ∈ R^n×c_x and B ∈ R^n×c_u.The control input is represented by v(u) = u.
  • Quadcopter example: The quadcopter model uses h = (R, p) ∈ SE(3) with four bidirectional-thrust control inputs.The state measurements include body-centered gravity, angular velocity, and linear velocity.
  • Quadcopter example: The quadcopter state is sampled at 200 Hz, and its basis includes nine products of body-centered linear and angular velocity components.The listed products include terms such as v3ω3, v2ω3, and ω1ω2.
  • Quadcopter example: The quadcopter stabilization controller uses a 0.1 s horizon and sets the additional-function weight Q̃ to zero.The appendix also specifies LQ weights and a regularization matrix for the stabilization problem.

C. Neural Network Automatic Function Discovery Configuration

The appendix gives configuration details for automatic function discovery and real-robot experiments. It specifies neural-network architectures, exploration schedules, controller weights, horizons, sampling rates, filtering, and state-function choices for simulated, SPRK, and Sawyer systems.

  • Automatic function discovery: The cart-pendulum and 2-link examples use three-layer networks with single hidden layers for zθ and vθ, producing c_x = 40 and c_u = 10.The cart-pendulum uses {4, 20, 40} and {2, 20, 10} layer sizes; the 2-link example uses {2, 20, 20} for vθ.
  • Automatic function discovery: Both simulated robot examples use 40% motor-saturation exploration noise that decays at 0.9^i+1, with information-weight decay at 0.2^i+1.The 2-link configuration uses a 0.05 s horizon and 100 Hz sampling rate.
  • SPRK configuration: The SPRK controller runs at 30 Hz, low-pass filters control vectors, uses information weight 80, and computes control over a 0.5 s horizon.Its controller weights are Q̃ = diag([60, 60, 5, 5, 1]) and R = diag([0.1, 0.1]).
  • SPRK configuration: SPRK active learning runs for 20 seconds before the information weight is set to zero for end-effector trajectory tracking.The appendix specifies the transition from learning to tracking explicitly.
  • SPRK configuration: The SPRK observable set uses a third-order polynomial expansion of velocity states, yielding an 18-dimensional function-observable vector.The listed vector includes position, velocity, constant, squared, and higher-order velocity terms.
  • Sawyer configuration: The Sawyer robot runs at 100 Hz with low-pass-filtered controls, seven torque inputs, information weight 2000, and a 0.5 s control horizon.Its state contains joint angles and velocities, and v(u) = u ∈ R^7.

APPENDIX B PROOFS

The appendix proves the sensitivity of a finite-duration controller switch using a mode insertion gradient. The derivation evaluates how replacing a nominal policy over a short interval changes the terminal objective through state-transition and adjoint variables.

  • Proposition 1: The mode insertion gradient gives the sensitivity of switching from μ to μ⋆ at time τ for an infinitesimally small duration λ.The result is stated for τ ∈ [t_i, t_i + T].
  • Proof: The proof models a trajectory generated by μ except during [τ, τ + λ], when controller μ⋆ is applied.The terminal state is therefore determined by a short policy replacement embedded within the nominal trajectory.
  • Proof: The dynamics map f(z, u) describes the time evolution of the Koopman state z(t).The proof treats f as a mapping from R^c_x × R^c_u to R^c_x.
  • Proof: Differentiating with respect to λ produces boundary terms f2 − f1 from applying Leibniz’s rule.Here f2 uses μ⋆(τ), while f1 uses the nominal policy μ(z(τ)).
  • Proof: A state-transition matrix propagates the instantaneous switching effect, and the adjoint variable provides the terminal-objective sensitivity.The adjoint uses terminal condition ρ(t_i + T) = ∂/∂z_m(z(t_i + T)).

B. Proof of Theorem 1

The proof derives a first-order expression for the information change produced by an active-learning controller relative to a task-minimizing controller. It introduces information measures for controllers, compares the corresponding objectives, and rearranges the result into a relative information-gain form.

  • Theorem 1 gives the first-order change in information under Assumption 1 and dynamics (20), subject to constraint (18).
  • The information terms Iµ⋆ and Iµ are identified as T-optimality measures associated with applying controllers µ⋆ and µ.
  • The proof defines controller-dependent information measures and distinguishes information acquired under a general control from information acquired under a controller.
  • The analysis compares the objective J(µ⋆) with J(µ), where µ minimizes the task loss ℓtask(z, u).
  • After applying the corollary, rearranging equation (38), inserting equation (21), and setting ∆I = Iµ⋆−Iµ, the proof obtains the relative information gain.
Loading 1906.05194v1…