Source-linked AI summary
From Pixels to Torques: Policy Learning with Deep Dynamical Models
Niklas Wahlström, Thomas B. Schön, Marc Peter Deisenroth
TL;DR
The paper addresses data-efficient closed-loop control from high-dimensional pixels when the true state is unavailable. It jointly learns a compact image representation and predictive dynamics, then uses adaptive MPC for control. In a pendulum task, the method reached almost 90% success after 15 trials, while the authors note that direct application of PILCO is impractical for many data points or high-dimensional problems.
Problem
Pixels-to-torques learning seeks continuous closed-loop policies from high-dimensional images without true-state access, while requiring good control after only a few trials.
Method
The method jointly learns a low-dimensional image embedding and a predictive forward model in feature space, using the resulting deep dynamical model with adaptive MPC.
Results
Almost 90% success after 15 trials was achieved by the proposed pixel-to-torque method, whereas PILCO on 2D auto-encoder and 20D PCA features failed consistently.
Takeaways & Limitations
Jointly capturing image representations and dynamics supports long-term predictions for data-efficient closed-loop control from pixels.
Takeaways & Limitations
Direct application of PILCO is impractical for many data points or high-dimensional problems, requiring adjustments for the pixels-to-torques setting.
Abstract
from arXiv · showhide
Data-efficient learning in continuous state-action spaces using very high-dimensional observations remains a key challenge in developing fully autonomous systems. In this paper, we consider one instance of this challenge, the pixels to torques problem, where an agent must learn a closed-loop control policy from pixel information only. We introduce a data-efficient, model-based reinforcement learning algorithm that learns such a closed-loop policy directly from pixel information. The key ingredient is a deep dynamical model that uses deep auto-encoders to learn a low-dimensional embedding of images jointly with a predictive model in this low-dimensional feature space. Joint learning ensures that not only static but also dynamic properties of the data are accounted for. This is crucial for long-term predictions, which lie at the core of the adaptive model predictive control strategy that we use for closed-loop control. Compared to state-of-the-art reinforcement learning methods for continuous states and actions, our approach learns quickly, scales to high-dimensional state spaces and is an important step toward fully autonomous learning from pixels to torques.
1. Introduction
The paper frames pixels-to-torques learning as the challenge of acquiring data-efficient closed-loop control from high-dimensional images without direct state access. It proposes combining learned low-dimensional representations, predictive dynamics models, and model-based control to address this setting.
- Motivation: Autonomous systems must process perceptual data, make decisions under uncertainty, and continually adapt without relying on specific human expert knowledge.The paper describes this as closing the perception-action-learning loop.
- Problem: The pixels-to-torques problem requires learning continuous-valued closed-loop policies from camera images alone while keeping the number of trials small.The robot has no direct access to joint configuration or other state measurements.
- Motivation: Many reinforcement learning algorithms require too many trials for real-world robots, motivating forward models for internal simulation and policy learning.Model-based approaches can improve data efficiency by learning the underlying dynamics before planning or control.
- Gap: Existing model-based methods often rely on heuristic or engineered low-dimensional features and do not readily scale to pixel-only learning because images have thousands of dimensions.This limitation motivates learning compact representations directly from observations.
- Approach: The proposed approach jointly learns a low-dimensional image embedding and a transition function, then uses predicted trajectories with adaptive model-predictive control for online closed-loop control.Its deep dynamical model combines deep auto-encoders for feature learning with a feedforward transition model in feature space.
- Approach: Unlike value-function-based model-based reinforcement learning methods, the approach applies model-based optimal control and is designed to scale to high-dimensional problems.Related visual model-based methods are distinguished by the use of low-dimensional features for fast predictions and online MPC learning.
2. Deep Dynamical Model
The deep dynamical model jointly learns a compact image representation and a nonlinear predictive model in feature space. It uses these components for fast long-term predictions and adaptive model-predictive control.
- Model structure: The DDM jointly embeds high-dimensional images into low-dimensional features and learns a predictive forward model in that feature space.The model uses deep auto-encoders for embedding and a feed-forward neural network for transitions.
- Model structure: The encoder maps observations y_t to features z_t, while the decoder reconstructs observations from those features with low reconstruction error.The intended representation is compact while preserving the image information needed by the model.
- Prediction model: The transition model predicts the next feature from an n-step history of past features and control inputs using a nonlinear NARX function.Predicted features are decoded into predicted high-dimensional observations for trajectory prediction.
- Training: The DDM is trained jointly by minimizing reconstruction and prediction errors over the encoder, decoder, and transition-model parameters.Including reconstruction error keeps predicted features consistent with encoded features, which supports multi-step prediction.
- Role in control: Fast predictions of high-dimensional observations through a low-dimensional embedded time series support the subsequent online feedback-control strategy.The paper uses these predictions within adaptive model-predictive control.
3. Learning Closed-Loop Policies from Images
The method uses DDM predictions within nonlinear MPC to control toward a reference image and updates the model online from newly collected trajectories. Adaptive exploration is needed because a poor initial model can drive the system away from useful regions.
- MPC on images: MPC predicts feature trajectories for candidate control sequences and applies only the first optimized control before replanning at the next observation.Repeated optimization turns the procedure into a closed-loop feedback strategy.
- MPC on images: For image-based MPC, the current state consists of present and past encoded features together with past control inputs.The DDM encoder makes this state available from current image observations.
- MPC on images: The controller minimizes predicted feature deviation from a reference feature while penalizing control magnitude.The reference feature is obtained by encoding the desired reference image.
- Learning from scratch: A randomly initialized model can steer the closed-loop system far from the desired reference, leaving unexplored regions without new data.This motivates explicit mechanisms for exploration or uncertainty handling during learning from scratch.
- Learning from scratch: Adaptive online MPC initializes the DDM with a random trial, selects mostly model-based controls with epsilon-greedy exploration, and updates the model after each trial.All collected data are added to the dataset before the next update.
4. Experimental Results
Experiments evaluate the deep dynamical model and its closed-loop controller on high-dimensional image sequences, including predictive modeling, feature structure, pendulum control, and comparisons with PILCO.
- Predictive model evaluation: The DDM predicts future tile frames accurately for both one-step and multiple-step horizons by concatenating one-step predictions.Long-term prediction is central to the model predictive control evaluation.
- Predictive model evaluation: Joint training produces a two-dimensional feature manifold encoding tile position, whereas sequential training yields unstructured features.The structured representation makes a relatively simple prediction model sufficient for the tile dynamics.
- Closed-loop policy learning: The learned pendulum controller improves gradually across trials and reaches the upright target in most cases after the 15th trial.Evaluation uses a greedy policy with ε = 0 and targets an angle of ±π.
- Closed-loop policy learning: Nearly 90% success is achieved after 15 trials with DDM+MPC, close to the ground-truth PILCO baseline, while PILCO with auto-encoder or PCA features fails consistently.Success means stabilizing the pendulum angle within 10° of the target; the DDM+MPC result uses 1500 image frames.
- Computational comparison: PILCO requires about 10 hours for 10 trials and 1000 data points in a 2D feature space, whereas DDM+MPC performs online MPC without significant computational overhead.The authors report that most computation in their method is spent learning the DDM.
5. Conclusion
The paper proposes a data-efficient model-based RL algorithm for learning closed-loop policies directly from pixels in continuous state and action spaces. It combines a deep dynamical model with model predictive control, with joint feature and prediction learning supporting high-quality long-term predictions.
- The algorithm learns closed-loop policies directly from pixel information in continuous state and action spaces.
- Its deep dynamical model performs long-term predictions in a compact feature space.
- An MPC controller uses those predictions to determine actions online without estimating a value function.
- Jointly learning the feature mapping and predictive model is crucial for capturing dynamic behavior and producing high-quality long-term predictions.
- Compared to state-of-the-art RL, the algorithm learns fairly quickly, scales to high-dimensional state spaces, and facilitates learning from pixels to torques.