Learning Plannable Representations with Causal InfoGAN

Thanard Kurutach; Aviv Tamar; Ge Yang; Stuart Russell; Pieter Abbeel

Source-linked AI summary

Learning Plannable Representations with Causal InfoGAN

Thanard Kurutach, Aviv Tamar, Ge Yang, Stuart Russell, Pieter Abbeel

arXiv:1807.09341v1Jul 24, 2018cs.LGcs.AIcs.CVcs.NEcs.ROstat.ML

TL;DR

The paper addresses how to generate goal-directed visual plans from high-dimensional observations without relying on manually specified representations. Causal InfoGAN learns a structured low-dimensional planning model whose transitions generate observation sequences, and demonstrates plausible rope-manipulation walkthroughs while noting an encoding limitation for real images.

Problem
Robots need to plan in unstructured environments, but learning plannable representations directly from high-dimensional observations remains difficult.
Method
Causal InfoGAN trains a generator conditioned on consecutive low-dimensional planning states and noise, using mutual information to make latent transitions informative about observations.
Results
The method generates convincing walkthrough sequences for rope manipulation from real robot image data and produces realistic manipulation plans.
Takeaways & Limitations
Structured latent spaces can support efficient planning while generating goal-directed trajectories from high-dimensional dynamical systems.
Takeaways & Limitations
For high-dimensional images, the learned Q(s|o) can classify generated observations accurately but classify real observations inaccurately because it is trained only on generated images.

Abstract

from arXiv · show

In recent years, deep generative models have been shown to 'imagine' convincing high-dimensional observations such as images, audio, and even video, learning directly from raw data. In this work, we ask how to imagine goal-directed visual plans -- a plausible sequence of observations that transition a dynamical system from its current configuration to a desired goal state, which can later be used as a reference trajectory for control. We focus on systems with high-dimensional observations, such as images, and propose an approach that naturally combines representation learning and planning. Our framework learns a generative model of sequential observations, where the generative process is induced by a transition in a low-dimensional planning model, and an additional noise. By maximizing the mutual information between the generated observations and the transition in the planning model, we obtain a low-dimensional representation that best explains the causal nature of the data. We structure the planning model to be compatible with efficient planning algorithms, and we propose several such models based on either discrete or continuous states. Finally, to generate a visual plan, we project the current and goal observations onto their respective states in the planning model, plan a trajectory, and then use the generative model to transform the trajectory to a sequence of observations. We demonstrate our method on imagining plausible visual plans of rope manipulation.

1 Introduction

The paper introduces Causal InfoGAN to learn low-dimensional, plannable representations from high-dimensional observations and generate goal-directed visual walkthroughs. It combines generative modeling with planning and demonstrates plausible rope-manipulation plans.

General-purpose robots need representations that support reasoning and action planning in unstructured environments such as homes or hospitals.
Classical AI planning benefits from abstract representations but often relies on human-designed state representations and perception.
Reinforcement learning can act from raw images, but high-dimensional reward design and reliance on instrumented rewards or demonstrations remain challenging.
Causal InfoGAN learns plannable representations for high-dimensional dynamical systems, using discrete models with graph search or continuous models with linear interpolation.
The generator combines random noise with consecutive low-dimensional planning states, while mutual-information maximization makes those states informative about observation transitions.
The learned model generates walkthroughs from initial to goal observations, and experiments show convincing rope-manipulation sequences from real robot image data.

2 Preliminaries and Problem Formulation

The paper formulates walkthrough planning as generating feasible intermediate observations rather than complete action sequences. It motivates this relaxation through temporal abstraction and introduces the GAN and InfoGAN concepts underlying the later framework.

2.1 Deep Generative Models based on GAN and InfoGAN: Deep generative models learn stochastic neural networks that approximate the observation distribution using a generator and discriminator.
2.1 Deep Generative Models based on GAN and InfoGAN: Vanilla GAN noise need not have interpretable structure, motivating InfoGAN's use of an additional state component and mutual-information objective.
2.1 Deep Generative Models based on GAN and InfoGAN: Mutual information measures how much knowing one random variable reduces uncertainty about another, and InfoGAN optimizes it through a variational posterior approximation Q(s|o).
2.2 Problem Formulation: The problem assumes a fully observable deterministic dynamical system with unknown transition function and trajectories collected under an arbitrary exploration policy.
2.2 Problem Formulation: Path planning seeks actions from a start observation to a goal, whereas walkthrough planning seeks a sequence of observations whose consecutive pairs are h-reachable.
2.2 Problem Formulation: The paper explicitly leaves the problem of how to best generate exploration data outside its scope.
2.2 Problem Formulation: Walkthrough planning replaces long-horizon action planning with waypoint observations, which can later guide short-horizon methods such as inverse dynamics or model-free RL.

3 Causal InfoGAN

Causal InfoGAN structures a sequential-observation generator around transitions in a low-dimensional latent planning system. Mutual-information training encourages the latent transitions to encode causally relevant changes while remaining compatible with efficient planning.

High-dimensional dynamics models face a trade-off between expressive transition modeling and efficient planning, which Causal InfoGAN addresses with structured representations.
The model is trained on sequential observation pairs, treating reachable next observations as the relevant causal structure for planning.
A latent planning system contains abstract states and a parametrized stochastic transition function that may differ from the real observation space.
The generator receives consecutive abstract states alongside noise, so the states represent transition-relevant features while noise captures less informative variation such as pixel details.
The Causal InfoGAN objective adds a mutual-information term to the GAN loss, encouraging the abstract model to capture salient changes induced in observations.
A variational lower bound with auxiliary classifier Q(s,s′|o,o′) makes the objective tractable and maps observation pairs to abstract-state pairs.
The mutual-information objective alone is insensitive to code-word ordering, so consistent meanings for consecutive abstract states are needed for sequential rollouts.

4 Planning with Causal InfoGAN models

Causal InfoGAN plans goal-directed trajectories by encoding observations into abstract states, searching in a learned transition model, and decoding the state path into observations. The framework supports discrete and continuous latent planning systems, with high-dimensional image encoding handled by latent-space search when the posterior is unreliable on real images.

4.1 General Planning Paradigm: Planning maps start and goal observations to abstract states, searches a feasible state trajectory with model M, and decodes it into observations.The three-stage paradigm encodes observations, plans through transition probabilities, and generates the corresponding observation sequence.
4.2 Encoding an Observation to a State: For high-dimensional images, latent-space search replaces Q-based encoding because Q can classify generated images accurately but real images inaccurately.The reported failure is attributed to training Q only on generated observations, allowing it to overfit to generated images; the simple search worked without GAN modifications.
4.3.2 Discrete Abstract States – Binary Representation: Binary states represent predicates as true-or-false properties, which Causal InfoGAN learns directly from data for more expressive abstract representations.The binary formulation is motivated by common AI-planning representations based on object properties.
4.3.2 Discrete Abstract States – Binary Representation: One-hot representations learn finite MDP transitions with implicit actions, whereas binary representations use explicit stochastic action vectors.In the binary model, an MLP maps state and random action vectors to independent binary distributions, with action marginalized in the transition probability.
4.3.3 Continuous Abstract States: Continuous latent planning encourages feasible trajectories by learning small local transitions, making linear interpolation between start and goal states likely to remain feasible.The model uses Gaussian perturbations with state-dependent diagonal covariance and adds an L2 covariance penalty to the loss.

5 Related Work

Prior approaches often depend on manually designed representations, task-specific rewards, or observation-space distances. Causal InfoGAN instead learns compact representations that capture causal structure and support planning.

Unlike SPTM’s observation-level graph, Causal InfoGAN learns a compact parametric planning model with potential advantages for scaling to more complex problems.
Many RL approaches require manually provided rewards or successful demonstrations, whereas this work learns a general dynamical model without a reward signal.
Classical AI planning often uses human-designed state representations and manually specified perception from observations.
Causal InfoGAN automatically extracts representations describing causal structure rather than relying on a prespecified skill set.

6 Experiments

Experiments evaluate Causal InfoGAN on synthetic navigation and real rope-manipulation data. It learns causal state abstractions, produces feasible walkthroughs, and outperforms baseline generative models in visual-plan evaluation.

Experimental setup: The experiments combine toy 2D navigation tasks with real robotic rope-manipulation images to evaluate state abstraction and visual planning.
2D navigation: Causal InfoGAN learns clusters aligned with domain dynamics, whereas Euclidean-distance baselines can group observations across disconnected regions.
2D navigation: In the rescaled door-key task, Causal InfoGAN learns the key predicate and its acquisition location, while standard clustering fails to separate key states.
2D navigation: The 2D planning evaluation reports average feasibility on held-out start/goal observations, with Causal InfoGAN achieving significantly better planning than baselines.
Rope manipulation: For rope manipulation, Causal InfoGAN generates walkthrough plans directly from data without additional human guidance, unlike prior plans requiring human-provided pose sequences.
Rope manipulation: A classifier-based trajectory score averaged over 57 task configurations shows that Causal InfoGAN achieves a significantly higher score than generative-model baselines.

7 Conclusion

Causal InfoGAN learns structured latent representations for sequential data and uses planning-compatible latents to generate goal-directed trajectories from high-dimensional dynamical systems.

Causal InfoGAN structures the latent space of sequential data to support efficient planning and goal-directed trajectory generation.
Realistic rope-manipulation plans suggest applications in robotics, where deformable-object manipulation is challenging.
Binary latent models connect deep representation learning with classical AI planning by learning object predicates directly from data.

A Algorithm

The algorithm trains a generative model, posterior, and transition model on sequential observations, while mutual-information and consistency objectives align latent transitions with observed dynamics.

Causal InfoGAN learns a planning-useful latent generative model from sequential observation pairs.
The generator produces paired observations from consecutive latent states and additional noise, while the discriminator is updated adversarially.
Mutual-information updates encourage generated observations to retain information about transitions in the latent planning model.
For continuous states, an additional update makes small local latent transitions generate plausible observations.
The optional self-consistency loss uses likely observation encodings to maximize likelihood of observed transitions under predicted states.
The self-consistency loss stabilized low-dimensional training but was not beneficial for high-dimensional observations because the posterior overfit generated samples and poorly predicted real states.

B.1 2D Navigation Experiment

The 2D navigation experiments used compact binary latent representations and domain-specific trajectory sampling, with noise added to stabilize the key-state representation.

Toy-domain models used four-dimensional latent states, three-dimensional actions, and four-dimensional noise in both key and tunnel domains.
Tunnel trajectories used short random-walk scales and sampling biased near the choke to increase room-to-room crossings.
The key domain required larger steps so particles could obtain the key and reach the door within one trajectory.
The key domain represented key possession with a binary variable and injected Gaussian noise because endpoint singularities otherwise hindered numerical training.
Both domains used two-layer perceptrons for generator, discriminator, and posterior, with a smaller two-layer transition network.
The latent representation was binary, and generator outputs were trained over one timestep without autoregression in the autoregressive submodule.

B.2 Rope Experiment

The rope experiment used sequential image pairs and compared Causal InfoGAN with baseline latent-code models under a Gaussian latent planning transition.

The rope implementation used standard DCGAN architectures and weighted the main generator loss more heavily than mutual-information and transition losses.
Causal InfoGAN doubled the noise dimension because its generator receives two abstract states, while the baselines used separate latent-code and noise dimensions.
The latent planning system used a uniform prior on [−1, 1] and a Gaussian transition with state-dependent diagonal variance.
Rope training used observation pairs separated by one timestep from the rope dataset of Nair et al.

B.2.1 Causal Classifier

The causal classifier evaluates whether image transitions are feasible, using sequential positive pairs and randomly sampled negatives from different rope-manipulation runs. It generalizes beyond its one-step training examples to assign smoothly decreasing feasibility as temporal separation grows.

Causal Classifier: The classifier is used both to post-select feasible observation-space transitions during planning and to score walkthrough trajectories.
Causal Classifier: The classifier is trained to distinguish feasible image transitions using one-timestep positive pairs and randomly sampled negatives from different rope-manipulation runs.Image preprocessing is used to prevent the classifier from exploiting background differences between runs.
Causal Classifier: Training and held-out test accuracy reach 100% and 98%, respectively.
Causal Classifier: For image pairs separated by more than one timestep, predicted feasibility decreases smoothly and monotonically as the temporal gap increases.The classifier assigns zero probability to pairs with large separations despite never training on transitions longer than one step.
Causal Classifier: The rope-image generator receives current and next seven-dimensional abstract states plus four-dimensional noise, producing the corresponding current and next observations.The discriminator processes two grayscale images, while the posterior predicts state means and variances from one image.