PaperlayerView exact revision on arXiv

Source-linked AI summary

State Representation Learning for Control: An Overview

Timothée Lesort, Natalia Díaz-Rodríguez, Jean-François Goudou, David Filliat

arXiv:1802.04181v2cs.AIcs.LGstat.ML

TL;DR

Robotics and control need compact, expressive representations of high-dimensional sensor data, but manually designing them is costly. This survey reviews SRL methods that learn low-dimensional states from interaction-related signals and generic objectives, synthesizing their approaches, applications, evaluation practices, and research directions. It concludes that SRL offers diverse objective-based strategies, while comparing representations and evaluating them robustly remain difficult, especially on real robots.

  • Problem

    Robotics control requires compact, informative representations of high-dimensional sensor data, while manually engineering suitable states is costly.

  • Method

    The survey reviews SRL approaches using reconstruction, dynamics, actions, priors, exploration, and other generic objectives, together with their implementations, applications, and evaluation methods.

  • Results

    The survey synthesizes commonalities and differences across state-of-the-art SRL objectives and discusses their use in simulated and real robotics control settings.

  • Takeaways & Limitations

    SRL provides multiple ways to learn compact control-relevant states from interaction-related information, but evaluation and comparison require careful interpretation.

  • Takeaways & Limitations

    Robust task-based evaluation often requires many policy runs and is feasible in simulation but seldom applicable on fragile real robots.

Abstract

from arXiv · show

Representation learning algorithms are designed to learn abstract features that characterize data. State representation learning (SRL) focuses on a particular kind of representation learning where learned features are in low dimension, evolve through time, and are influenced by actions of an agent. The representation is learned to capture the variation in the environment generated by the agent's actions; this kind of representation is particularly suitable for robotics and control scenarios. In particular, the low dimension characteristic of the representation helps to overcome the curse of dimensionality, provides easier interpretation and utilization by humans and can help improve performance and speed in policy learning algorithms such as reinforcement learning. This survey aims at covering the state-of-the-art on state representation learning in the most recent years. It reviews different SRL methods that involve interaction with the environment, their implementations and their applications in robotics control tasks (simulated or real). In particular, it highlights how generic learning objectives are differently exploited in the reviewed algorithms. Finally, it discusses evaluation methods to assess the representation learned and summarizes current and future lines of research.

1 Introduction

State representation learning seeks compact, informative features of high-dimensional sensor observations for control, especially in robotics. This survey reviews unsupervised or weakly supervised approaches that exploit observations, actions, rewards, and generic objectives.

  • Motivation: Low-dimensional states compress high-dimensional sensor data while retaining information relevant to a robot’s task.Robotics often produces camera or other sensor observations whose useful control variables occupy a much smaller space.
  • Motivation: Low-dimensional, informative representations can make reinforcement-learning and other control algorithms solve tasks more efficiently than using raw observations.This efficiency matters when collecting robot experience is costly.
  • Learning setting: SRL aims to learn features with little or no direct supervision by using actions, their observational consequences, rewards, and generic constraints.The paper frames model building, rather than ordinary pattern classification or regression, as the relevant learning process.
  • Survey scope: The review analyzes recent approaches that learn low-dimensional Markovian representations from sequences of observations, actions, rewards, and generic learning objectives.Most surveyed methods are evaluated in simulation, while some are tested on real robotics tasks such as manipulation or exploration.
  • Survey scope: The paper organizes the survey around formal notation, learning objectives, implementation aspects, and current and future research directions.It also highlights commonalities and differences among approaches.

2 Formalism and definitions

The paper formalizes SRL as learning a compact state from sensor history without access to the true state, using proxy objectives that encode reconstruction, dynamics, actions, or prior knowledge. It then characterizes desirable representations as Markovian, useful for value estimation and generalization, low-dimensional, disentangled, sufficient, efficient, and minimal, while noting that no unique solution is required.

  • 2.1 SRL Formalism: In the formalism, an agent acts in an environment with unknown true states, receives sensor observations, and may receive task rewards.Actions can be continuous or discrete, and rewards are optional for representation learning.
  • 2.1 SRL Formalism: SRL learns a K-dimensional representation st from observation history, optionally incorporating past actions and rewards, without using the true state.The representation is produced by a mapping from observation history to the current state.
  • 2.2 SRL approaches: Reconstruction methods encode observations into low-dimensional states and decode them back, minimizing reconstruction error under constraints such as dimensionality, denoising, or sparsity.Encoder and decoder parameters are learned separately within the reconstruction model.
  • 2.2 SRL approaches: Forward models predict the next state from the current state or observation and action, back-propagating next-state prediction error through transition and encoding models.Structural constraints, such as linear dynamics in the learned state space, can be imposed on the forward model.
  • 2.2 SRL approaches: Inverse models predict the action connecting consecutive states, training the encoder so the learned state retains information sufficient to recover the action that caused the transition.The action-prediction error is back-propagated through the encoding pathway.
  • 2.2 SRL approaches: Prior-based methods apply loss functions to sets of states under conditions such as locality or temporal proximity to impose constraints on the learned state space.These objectives encode assumptions about functioning, dynamics, or world physics.
  • 2.3 State representation characteristics: A desirable state representation is Markovian, supports value estimation and generalization to similar futures, and remains low-dimensional for efficient estimation.These properties are evaluation criteria rather than direct learning objectives.
  • 2.3 State representation characteristics: Disentangled representations separate factors of variation, while sufficiency, efficiency, and minimality favor compact representations that exclude irrelevant information.The paper notes that multiple representations can satisfy the desired properties, so SRL has no unique solution.

3 Learning objectives

SRL methods use reconstruction, temporal prediction, action inference, and other constraints to learn compact representations without direct access to the true state. Many approaches combine objectives so representations capture both observation content and system dynamics.

  • Reconstructing the observation: Observation reconstruction learns low-dimensional states by encoding observations and decoding them while imposing dimensionality or other representation constraints.Auto-encoders reconstruct the input, whereas dynamic variants reconstruct observations from other time steps or constrain state evolution.
  • Reconstructing the observation: Static auto-encoders may fail when observation variation is not linked to the actual state, motivating temporal constraints that emphasize system-relevant features.The survey describes this limitation and introduces dynamic reconstruction as one way to relax the assumption.
  • Learning a forward model: Dynamic auto-encoder approaches include siamese encoders, linear-transition constraints, deep generative models, and on-policy learning for visual or tactile control.E2C learns image trajectories from a linear latent space, while related methods embed dynamic constraints during policy learning.
  • Learning a forward model: Forward models learn representations by predicting the next state or observation from the current representation and action, with prediction error backpropagated through the encoder.Some methods assume linear transitions in the learned state space, then decode predicted states into future observations for pixel-wise reconstruction.
  • Learning a forward model: Forward-model objectives can avoid incentivizing representations to encode uncontrollable features, supporting robustness to distractors, illumination changes, and environmental noise.This claim is presented in the context of an intrinsic-curiosity model for exploration.
  • Learning an inverse model: Inverse models can supervise representation learning while forward models regularize it by predicting the next state; combining both losses often improves learned representations.An ablation study identifies the inverse model as the largest contributor to representation quality among the tested components.
  • Using hybrid objectives: In object-pushing simulations, inverse or jointly inverse-forward models improve performance, and the joint model remains comparable to larger-data training when data are reduced.The reported comparison concerns interactively learning poke actions from current and goal images.

4 Building blocks of State Representation Learning

SRL implementation and evaluation involve surrogate models, loss-function tools, and strategies for constraining the information bottleneck and generalizing low-dimensional representations.

  • Building blocks of State Representation Learning: The section covers surrogate models, loss-function specification tools, and strategies for constraining information bottlenecks and improving generalization.These implementation aspects are presented as relevant to both state representation learning and its evaluation.

4.1 Learning tools

Learning tools provide auxiliary objectives for learning compact state representations, including reconstruction, probabilistic encoding, and constraints across multiple states.

  • Auxiliary objective functions can be integrated into broader state representation learning approaches.
  • Auto-encoders: Auto-encoders learn compact representations by encoding observations into a latent space and minimizing reconstruction error.Mean squared error is usual, but any norm can be used.
  • Auto-encoders: PCA can be considered a particular case of an auto-encoder.
  • Denoising auto-encoders: Denoising auto-encoders add input noise to reduce the average-image solution associated with pixel reconstruction.In one comparison, DAE representations gathered less reward than VAE representations in most cases.
  • Variational auto-encoders: Variational auto-encoders use probabilistic hidden cells to model a distribution over latent state representations.
  • Siamese networks: Siamese networks share parameters across identical branches and impose latent-space constraints among multiple states.They can encode similarity or temporal dependencies, including slowness between consecutive states.

4.2 Observation/action spaces

The reviewed approaches span environments with varied observation, action, and state dimensions, while choosing an appropriate state dimensionality remains difficult.

  • Higher observation and action dimensions, together with smaller desired state dimensions, increase the difficulty of learning state representations.More information must be processed and filtered to retain useful content.
  • Reviewed studies often use learned states with higher dimensionality than theoretically necessary.The survey gives a 6-dimensional state for a 2-joint robotic arm as an example.
  • Learning objectives are proxies that can produce representations differing from the ideal minimal state.
  • Increasing state dimensionality above the true-state dimension can improve optimization of the learning objectives.
  • The survey’s settings table summarizes environment characteristics and state-representation dimensions across approaches.

4.3 Evaluating learned state representations

Learned state representations are evaluated through geometry, neighborhood coherence, disentanglement, prediction of ground-truth features, and downstream control performance.

  • The survey summarizes validation metrics and embedding-quality techniques for assessing learned state spaces.
  • Control-task evaluation tests whether an agent can use the representation to learn a transferable policy.
  • Control-task evaluation is costly and depends on the reinforcement-learning algorithm, so it is sufficient but not necessary for validating a representation.
  • Nearest-neighbor inspection compares observations close in learned space with their proximity in the intended ground-truth state space.
  • KNN-MSE quantitatively measures ground-truth distances between an observation and neighbors retrieved in learned state space.
  • Disentanglement evaluation includes selectivity and a classifier-based disentanglement metric, with assumptions about action or generative-factor independence.
  • Distortion and NIEQA assess global or local geometric and topological coherence relative to the original space.
  • Supervised regression from representations to ground-truth states evaluates whether encoded features generalize to unseen states.

4.4 Evaluation scenarios

SRL methods are evaluated across varied benchmarks, mainly simulated environments, alongside robotics tasks ranging from manipulation to balancing a real pendulum.

  • SRL datasets are varied and mainly simulated because simulated environments are easier to reproduce and generate.
  • Common benchmarks include pendulum, cart-pole, Atari, VizDoom, and Mario environments.
  • Other evaluations use simulated octopus arms, labyrinths, navigation grids, driving cars, mountain cars, and bouncing-ball scenarios.
  • Robotics benchmarks cover manipulation skills such as pushing, grasping, stabilizing, poking, and balancing a real pendulum.
  • Many simulated scenarios belong to Universe, OpenAI Gym, or DeepMind Labs.

5 Discussion and future trends

The discussion identifies exploration, dimensionality selection, evaluation, comparison, reproducibility, and interpretability as central challenges and research directions for SRL. It also highlights simulation-to-real transfer and the need for meaningful explanations of learned representations.

  • SRL models for autonomous agents: SRL requires environment exploration because insufficient coverage can limit varied observations and exposure to actions associated with optimal performance.Curiosity and intrinsic motivation can augment extrinsic rewards to encourage exploration, but representation-focused automatic exploration remains an open direction.
  • SRL models for autonomous agents: Simulation data may provide relevant augmentation, but the reality gap constrains how models trained with simulation benefit real-world SRL.Combining simulated and real data has improved results in some applications, motivating further study of simulation data for real-world representations.
  • SRL models for autonomous agents: Most reviewed approaches choose representation dimensionality empirically, while automatic selection must balance model capacity against training-error reduction and overfitting.PCA-based feature selection offers one alternative, although its variance threshold still must be fixed beforehand.
  • Assessment, comparison and reproducibility in SRL: SRL assessment lacks a certified validation method and common evaluation frameworks, making fair comparison between approaches difficult.Published results vary substantially across environments and datasets, supporting evaluation frameworks with multiple tasks and metrics plus reproducibility procedures.
  • Assessment, comparison and reproducibility in SRL: Task performance with an RL controller is the most objective representation-quality test, but stochasticity, algorithm choice, and real-robot constraints make robust assessment costly.Multiple policy-evaluation runs are feasible in simulation but seldom applicable on fragile real robots, so partial ground-truth-based measures can complement them.
  • Interpretability: SRL interpretability links representation variation to environmental variation and the reason for sensitivity, while higher representation dimension reduces human interpretability.Meaningful explanations require interpretability to be properly defined and monitored.

6 Conclusion

The conclusion presents SRL as a way to compress high-dimensional observations into low-dimensional, meaningful spaces using observations, actions, and optionally rewards. The survey synthesizes learning objectives, evaluation tools, benchmarks, comparison challenges, interpretability, and future directions including continual learning and exploration-aware data collection.

  • 6 Conclusion: SRL compresses high-dimensional observation data into a low-dimensional, meaningful space for controlled systems.The models use agent observations and actions, with task rewards optionally included.
  • 6 Conclusion: The survey provides an accessible guide to SRL approaches, evaluation tools, and common simulation benchmark settings.It also compares state-of-the-art learning objectives and discusses autonomous agents, approach comparison, and interpretability.
  • 6 Conclusion: Combining reconstruction, predictive, reward-based, and broader prior objectives can connect observations, dynamics, action effects, and human-understandable state structure.No proposed model currently integrates all of these objectives together.
  • 6 Conclusion: Future SRL work could address evolving environments as a step toward continual or lifelong learning and design exploration strategies specifically to improve learned representations.These directions concern learning from unlabeled observations and collecting data that benefits representation quality.
Loading 1802.04181v2…