Deep Learning of Koopman Representation for Control

I. INTRODUCTION

• Choosing suitable finite basis functions for Koopman lifting remains a central challenge; standard choices include radial basis and polynomial functions.The paper notes that these a priori choices do not take advantage of known physics or data when selecting the basis.Source: §I · p. 1
• The paper extends DNN-based identification of Koopman basis functions from autonomous systems to controlled dynamical systems.The proposed learning algorithm automatically searches for suitable basis functions to construct an approximated linear model in the lifted space.Source: §I · p. 1
• OpenAI Gym is used to generate data and train the DNN, with example systems demonstrating the framework's capability.The paper presents the work as a step toward bridging model-based control and model-free reinforcement learning.Source: §I · p. 1

II. KOOPMAN LEARNING USING DEEP NEURAL NETWORK

• DNNs learn basis functions for a finite-dimensional linear lifting of nonlinear control dynamics, enabling linear-system control methods such as LQR and MPC.The paper chooses linear lifting for computational focus, although control-affine systems can have bilinear lifts.Source: §II · p. 2
• The approximation is non-convex because basis functions and linear system matrices are identified simultaneously.Source: §II · p. 2
• Training pairs state observations and their successors; the DNN outputs lifting functions learned directly from the training data.The loss is designed to enforce both the Koopman transformation and linearity.Source: §II · pp. 2–3
• The approach keeps A and B fixed during each training epoch, then updates them analytically between epochs.The described update uses a learning rate of 0.5; a separate optimization computes C to map lifted states back to state space.Source: §II · p. 3

III. KOOPMAN-BASED CONTROL

• The lifted linear system is used to solve a discrete LQR problem with a quadratic cost.Source: §III · p. 3
• MPC can accommodate problem constraints by optimizing a sequence of control inputs over a finite horizon.Source: §III · p. 3
• Spectral analysis approximates Koopman eigenfunctions using the identified A matrix to interpret the generated control strategy.Source: §III · p. 3
• The algorithm trains the neural network and obtains system matrices before applying LQR or MPC with constraints.The algorithm also specifies a goal position and constructs a goal-relative lifted state.Source: §III · p. 4
• The learned pendulum eigenfunctions have a similar double-pole structure in uncontrolled and controlled cases, with their distributions reflecting energy-related state variation.The paper notes that joint torque enters the controlled system’s energy accounting.Source: §III · p. 4

• The method is demonstrated with MPC on Inverted Pendulum and LQR on Lunar Lander (Continuous Control).Source: §IV · p. 4
• Figure 3 depicts the inverted pendulum simulation.Source: Figure 3 · p. 4
• Figure 4 shows real and imaginary parts of learned controlled-pendulum eigenfunctions; the dominant eigenfunction has zero imaginary value.Source: Figure 4 · p. 4

• OpenAI Gym's Box2D engine provides the simulation dynamics, with gravity, object interaction, and physical contact configurable in the world file.Source: A. Testing Environment · p. 4
• The model does not know the simulation dynamics in advance and must learn them from training data.Source: A. Testing Environment · p. 4
• Each game is randomly initialized to generate an initial disturbance.Source: A. Testing Environment · p. 4

• DDPG is the state-of-the-art reinforcement-learning algorithm used as the comparison method.Source: B. Reinforcement Learning · p. 4
• Direct side-by-side comparison of learning processes is difficult because DKRC trains on segmented data pairs from a batch, while RL uses time-series data streams.Source: B. Reinforcement Learning · p. 4
• Comparison with controllers using predetermined EDMD or DMD basis functions is deferred to future work.Source: B. Reinforcement Learning · p. 4

• The pendulum starts at a downward orientation and must reach (θ = 0, θ̇ = 0); its recorded angle uses cosine and sine values for smooth transitions at −π and π.Source: C. Inverted Pendulum · p. 5
• The governing dynamics are unknown and must be recovered from limited time-series data; restricted velocity and torque can require swing-back motion to build momentum.Source: C. Inverted Pendulum · p. 5
• DDPG leaves failed trails, whereas DKRC minimizes system energy within a short deployment time.Source: Figure 5 · p. 5, C. Inverted Pendulum · p. 5
• Figure 6 shows ten DKRC trajectories from random initializations, color-mapped by Hamiltonian energy, with goal states marked in red.Source: Figure 6 · p. 5

D. Lunar Lander

• The Lunar Lander environment has six state dimensions and two control inputs, with main-engine input u1 in [0, 1] and side-engine input u2 in [−1, 1].The main engine takes only positive input; the side engine's signed input selects left or right firing.Source: D. Lunar Lander · p. 5
• DKRC training used 1,876 data pairs from five games, collected with an untrained reinforcement-learning algorithm using a random policy and Ornstein–Uhlenbeck noise.Source: D. Lunar Lander · p. 5
• Because the exact Gym system model cannot be extracted directly, it must be identified through a data-driven model-based or model-free approach.Source: D. Lunar Lander · p. 5
• 10 DKRC games returned to the landing goal despite spread-out trajectories from randomized initial velocities.The initial velocities vary between games, and the control offsets drift while keeping the lander balanced.Source: D. Lunar Lander · p. 6, Figure 8 · p. 6
• After the Koopman transform, the dynamics are assumed linear so model predictive control can be applied.Source: D. Lunar Lander · p. 6
• The game score rewards a smooth landing in the flagged zone while penalizing engine fuel consumption.Source: D. Lunar Lander · p. 6

V. CONCLUSIONS

• DKRC learns in a completely data-driven way while keeping control design model-based.Source: §V · p. 6
• The authors report efficient training and sufficient deployment in two OpenAI Gym environments compared with DDPG.Source: §V · p. 6