Source-linked AI summary
Visual Semantic Planning using Deep Successor Representations
Yuke Zhu, Daniel Gordon, Eric Kolve, Dieter Fox, Li Fei-Fei, Abhinav Gupta, Roozbeh Mottaghi, Ali Farhadi
TL;DR
Visual semantic planning asks how an agent can turn visual observations into action sequences that reach goals in a dynamic environment. The paper learns object and action knowledge through interaction, bootstraps reinforcement learning with imitation, and uses deep successor representations. It reports near-optimal performance across varied THOR tasks and promising cross-task transfer.
Problem
Visual semantic planning requires an agent to predict actions from visual observations while learning object affordances and action preconditions and effects.
Source: §1 · p. 1Method
The paper learns object and action knowledge through environmental interaction, bootstraps reinforcement learning with imitation learning, and uses successor representations to support cross-task transfer.
Source: §1 · p. 2Results
Takeaways & Limitations
The results support interaction-centric visual planning with learned representations of object affordances and action preconditions and effects in THOR.
Source: §6 · p. 8The stated transfer result is conditioned on a bound on successor-feature approximation error.
Source: §4.5 · p. 6
Abstract
from arXiv · showhide
A crucial capability of real-world intelligent agents is their ability to plan a sequence of actions to achieve their goals in the visual world. In this work, we address the problem of visual semantic planning: the task of predicting a sequence of actions from visual observations that transform a dynamic environment from an initial state to a goal state. Doing so entails knowledge about objects and their affordances, as well as actions and their preconditions and effects. We propose learning these through interacting with a visual and dynamic environment. Our proposed solution involves bootstrapping reinforcement learning with imitation learning. To ensure cross task generalization, we develop a deep predictive model based on successor representations. Our experimental results show near optimal results across a wide range of tasks in the challenging THOR environment.
1. Introduction
Visual semantic planning asks an agent to turn visual observations into an action sequence that reaches a goal, requiring knowledge of objects, actions, and how actions change the scene. The paper learns this knowledge through interaction and combines imitation learning with reinforcement learning and successor representations to support planning and transfer.
1. Introduction
- A successful visual plan must sequence actions that transform an initial scene into a goal state, such as finding and moving a bowl before placing it in a microwave.Actions must satisfy one another’s preconditions for the sequence to succeed.Source: §1 · p. 1
- Visual planning requires learning object affordances and action preconditions and effects, while coping with costly exploration, action sequencing, and cross-task transfer.The paper notes that visual variability can make naive exploration intractable.Source: §1 · p. 1
- The paper treats visual semantic planning as policy learning over high-level actions, excluding low-level motor control and motion planning.Its interaction-centric approach learns object and action knowledge by interacting with a visual dynamic environment, using an extended THOR framework.Source: §1 · pp. 1–2
- Imitation learning bootstraps reinforcement learning for large action spaces and delayed rewards, while successor representations decouple dynamics from task rewards for transfer to new tasks.The paper describes theoretical guarantees for this transfer.Source: §1 · p. 2
- The proposed model achieves near-optimal results across varied tasks in THOR, with reported cross-task transfer and learned features encoding object affordances and action effects.The authors also report transferability properties and qualitative evidence that the representation encodes objects, actions, and environments.Source: §1 · p. 2
2. Related Work
Related work spans formal task planning, perception learned through interaction, deep reinforcement learning, and simulated visual environments. The paper positions its contribution at their intersection: interactive visual policies with imitation-bootstrapped successor representations and cross-task transfer.
Task planning
- Task-planning methods typically use formal languages and low-dimensional states, whereas visual semantic planning faces high-dimensional, partially observable input and seeks cross-task generalization.The paper contrasts its generalization aim with methods usually designed for a specific environment and task.Source: §2 · p. 2
Perception and interaction
- Interactive perception research has pursued faster learning and more robust visual representations, while environment interaction has also supported work on physical understanding [39]; [9] [26] [28] [36].The authors frame their work as integrating perception and interaction through policies mapping pixels to actions.Source: §2 · p. 2
Deep reinforcement learning
- Prior deep RL includes model-free and model-based approaches, while successor representations combine aspects of both and support value-function transfer; [3]; [24].This paper proposes a reduced-parameter successor architecture, trained first with imitation learning and then fine-tuned with reinforcement learning.Source: §2 · p. 2
Synthetic data for visual tasks
- Simulated platforms have supported visual-task training and robotics learning; THOR provides realistic indoor scenes, and this work extends it with object actions and a planner [54].The passage also situates simulation as a setting for learning and testing before real-world deployment.Source: §2 · p. 2
3. Interactive Framework
The interactive framework extends THOR for object interaction and represents scenes, object actions, and their rules for planning. It uses semantic actions and a STRIPS formulation to make visual planning problems explicit.
3. Interactive Framework
- The framework extends THOR with object states and a discrete scene description in a planning language.Figure 2 illustrates visual state changes for the framework’s action types, including actions that enable later interactions.Source: §3 · pp. 2–3, Figure 2 · p. 3
3.1. Scenes
- The extended framework contains 10 kitchen scenes, each averaging 53 interactable objects.The scenes were developed using the Unity 3D game engine.Source: §3.1 · p. 3
3.2. Objects and Actions
- Objects are divided into movable items and stationary receptacles, with containers requiring an open door for placement and the agent limited to holding one item.The framework defines six action types: navigate, open, close, pick up, put, and look up or down.Source: §3.2 · p. 3
- The action set averages 80 actions per scene, abstracts away navigation and continuous motion, and leaves action preconditions and effects for the agent to learn through interaction.For example, an item cannot be placed in a closed fridge; the rules are encoded in the environment but not exposed to the agent.Source: §3.2 · p. 3
3.3. Planning Language
- The framework uses STRIPS to represent a planning problem through an initial state, goal condition, and actions with preconditions and postconditions.This formulation expresses scene rules, including object affordances and action causality.Source: §3.3 · p. 3
4. Our Approach
The approach frames visual semantic planning as policy learning, using imitation learning to bootstrap a successor-representation model and reinforcement learning to improve it.
4.1. Successor Representation
- The successor representation factors value into a reward predictor and predictive successor features, separating task rewards from policy-dependent environment dynamics.The successor features summarize expected future feature occupancy; the reward vector represents the task.Source: §4.1 · p. 4
4.2. Our Model
- The model represents visual planning as finding a policy that reaches a Boolean-defined goal, and predicts reward and Q values from state-action features.A convolutional image encoder and internal-state and action encoders feed separate state-action-feature and successor-feature branches.Source: §4.2 · p. 5
4.3. Imitation Learning
- Imitation learning bootstraps the model with planner trajectories, remapping planner states to image-action pairs and adding random off-plan actions before recomputing trajectories.The training objective combines reward-prediction and Q-prediction losses.Source: §4.3 · p. 5
4.4. Reinforcement Learning
- During reinforcement learning, reward prediction remains supervised while a Bellman-based objective trains successor features; updates alternate between branches using replay-buffer minibatches.The successor-feature target is learned iteratively without expert trajectories.Source: §4.4 · pp. 5–6
4.5. Transfer with Successor Features
- For a new task with unchanged scene dynamics, the model freezes all parameters except the reward vector w, enabling more sample-efficient policy learning than training from scratch.The paper grounds this transfer approach in a theoretical performance bound for successor features.Source: §4.5 · p. 6
4.6. Implementation Details
- Implementation trains with imitation learning for 500k iterations, then fine-tunes with reinforcement learning for 10,000 episodes.Imitation learning uses batch size 32 and learning rate 1e-4, and includes the successor loss.Source: §4.6 · p. 6
5. Experiments
The experiments evaluate successor-representation policies across household tasks, task transfer, and affordance learning. The reported results show SR methods outperforming baselines across task difficulties, with additional evidence of adaptation and action-outcome discrimination.
5.1. Quantitative Evaluation
- Imitation learning followed by RL fine-tuning is evaluated against RL-only and behavior-cloning baselines, including a memory-equipped classifier.The comparison also includes random-action agents; RL from scratch cannot handle THOR’s large action space.Source: §5.1 · p. 6
- Tasks span three planning difficulties; hard tasks randomize agent and item locations and require memory of previously searched receptacles.Evaluation uses 10 easy, 8 medium, and 7 hard tasks, each across 100 episodes; hard-task models imitate a fixed-order search plan and receive no RL fine-tuning.Source: §5.1 · pp. 6–7
- Across easy, medium, and hard tasks, SR methods outperform baselines, and RL fine-tuning further reduces steps toward the goal.Table 1 covers all three difficulties; the SR IL model is not fine-tuned for hard tasks.Source: §5.1 · p. 7, Table 1 · p. 7
5.2. Task Transfer
- Fine-tuning reward vector w adapts a trained SR policy to new items in the same scene faster than training from scratch or fine-tuning the whole model.The transfer experiments substitute lettuce, egg, container, and apple for a bowl-search task; adapting w yields high episode success and successful-action rates.Source: §5.2 · pp. 7–8, Figure 5 · p. 7
5.3. Learning Affordances
- Imitation learning produces fewer failed actions than standard RL, while its supervised loss discourages actions that do not move toward the goal.The affordance evaluation does not explicitly label action success or failure during the tasks.Source: §5.3 · p. 8, Figure 6 · p. 8
- After random-policy interaction with reward +1.0 for successful actions and −1.0 for unsuccessful ones, the SR network reaches 0.91 ROC-AUC on immediate-reward prediction.Its state-action features cluster successful and unsuccessful pairs separately in the t-SNE visualization.Source: §5.3 · p. 8, Figure 7 · p. 8
6. Conclusions
The deep successor-representation model achieves near-optimal results across varied THOR tasks and shows promising cross-task transfer. Its learned successor features encode object affordances and action preconditions and post-effects.
6. Conclusions
- The deep successor-representation model achieves near-optimal results across tasks of varying difficulty in the THOR environment.The authors also report promising cross-task knowledge transfer.Source: §6 · p. 8
- Learned successor features encode object affordances and action preconditions and post-effects.The authors identify this as a qualitative result.Source: §6 · p. 8
- The authors propose exploring transfer from THOR to real-world environments and tasks with richer action sets as next steps.Source: §6 · p. 8
A.1. Experiment Setup
The experiments use Adam to train the successor-representation model and specify reinforcement-learning settings and reward schemes for the task difficulty levels.
A.1. Experiment Setup
- The successor-representation model is trained with Adam at a 1e-4 learning rate and mini-batches of 32.Source: §A.1 · p. 11
- Reinforcement learning uses γ = 0.99, a 100,000-entry replay buffer, and ε-greedy evaluation with ε = 0.1.During training, ε is annealed from 1.0 to 0.1; soft target updates with τ = 0.1 occur after each episode.Source: §A.1 · p. 11
- Easy and medium tasks reward completion (+10.0), penalize invalid actions (−5.0), and penalize other actions (−1.0).Source: §A.1 · p. 11
- For hard tasks, the model imitates a fixed receptacle-visitation order; deviating from it ends the episode in failure.Completion earns +1.0 immediate reward.Source: §A.1 · p. 11
A.2. Network Inputs
The model input combines the action, first-person visual observation, and the agent’s internal state, each encoded for the successor-representation model.
A.2. Network Inputs
- Actions combine a 7-dimensional one-hot action-type vector with an object-argument vector, including a null option for Look Up and Look Down.The seven types are Navigate, Open, Close, Pick Up, Put, Look Up, and Look Down.Source: §A.2 · p. 11
- Visual input is four stacked 84 × 84 grayscale frames from the agent’s first-person camera.RGB images are preprocessed to grayscale.Source: §A.2 · p. 11
- Internal state encodes inventory, agent rotation, and camera viewpoint as one-hot vectors.Rotation has four 90-degree orientations, and viewpoint has three tilting angles: −30°, 0°, and 30°.Source: §A.2 · p. 11
A.3. Network Architecture
The architecture separately encodes visual input, actions, and internal state, then predicts state-action and successor features. Training uses replay and a target network to stabilize learning.
A.3. Network Architecture
- The image encoder maps an 84 × 84 × 4 input through three convolutional layers to a 512-dimensional feature.The layers use 32 filters of size 8 × 8, 64 of size 4 × 4, and 64 of size 3 × 3, with strides 4, 2, and 1.Source: §A.3 · p. 11
- Two-layer MLP encoders process actions and internal state, while two further MLPs produce 512-dimensional state-action and successor features.A reward-predictor vector is dotted with these features to compute immediate rewards and Q values; hidden layers use ReLU.Source: §A.3 · p. 11
B. Algorithm Details
- The reinforcement-learning procedure follows deep Q-learning and deep successor-representation methods, using replay buffers and a target network for training stability.Source: B. Algorithm Details · p. 11
C. Action Space
The action space reflects scene objects and their affordances, with simplifications to reduce its size and speed learning.
C. Action Space
- On average, each scene has 53 objects, some interactable, and supports 80 agent actions.The set of plausible actions depends on the variety of objects in the scene.Source: C. Action Space · p. 11
- Scene #9 has 16 items, 23 receptacles at 11 locations, and 15 containers; Figure 8 shows the scene.Its 80 actions include 11 navigation, 15 open, 15 close, and 14 pick-up actions.Source: C. Action Space · p. 11, Figure 8 · p. 11
- Navigation destinations and same-category pickups are merged, and navigation actions are treated as teleports; PDDL checks action preconditions before execution.The teleport abstraction leaves visual navigation to an independent subroutine described in previous work [54].Source: C. Action Space · p. 12
D. Tasks
The evaluation covers 25 tasks across three difficulty levels.
D. Tasks
- The experiments evaluate 10 easy tasks, 8 medium tasks, and 7 hard tasks.The task list is provided in Table 2.Source: D. Tasks · p. 12