Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Image

Danny Driess; Jung-Su Ha; Marc Toussaint

Source-linked AI summary

Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Image

Danny Driess, Jung-Su Ha, Marc Toussaint

arXiv:2006.05398v1Jun 9, 2020cs.LGcs.AIcs.CVcs.ROstat.ML

TL;DR

TAMP must coordinate symbolic action sequencing with continuous motion planning, but combinatorial action spaces make conventional search costly. The paper uses a goal-conditioned convolutional recurrent network to predict action sequences from scene images, integrates it with TAMP search, and reports that the first prediction is feasible in most cases while generalizing beyond two-object training scenes. The method assumes the initial image contains enough information to resolve the task.

Problem
Varying-object scenes and large action-sequence spaces make it difficult to learn useful TAMP predictions without retaining costly search and motion-planning evaluations.
Method
A convolutional recurrent network encodes scene objects and goals in images, predicts goal-conditioned action sequences, and integrates them into the TAMP tree search.
Results
In most cases, the first network-generated sequence was feasible, and the representation generalized to multiple-object scenes despite training on only two objects at a time.
Takeaways & Limitations
The approach can often avoid discrete search and solve the resulting TAMP instance with only one trajectory-optimization problem.
Takeaways & Limitations
The initial scene image must contain sufficient information, with no total occlusions or other ambiguities.

Abstract

from arXiv · show

In this paper, we propose a deep convolutional recurrent neural network that predicts action sequences for task and motion planning (TAMP) from an initial scene image. Typical TAMP problems are formalized by combining reasoning on a symbolic, discrete level (e.g. first-order logic) with continuous motion planning such as nonlinear trajectory optimization. Due to the great combinatorial complexity of possible discrete action sequences, a large number of optimization/motion planning problems have to be solved to find a solution, which limits the scalability of these approaches. To circumvent this combinatorial complexity, we develop a neural network which, based on an initial image of the scene, directly predicts promising discrete action sequences such that ideally only one motion planning problem has to be solved to find a solution to the overall TAMP problem. A key aspect is that our method generalizes to scenes with many and varying number of objects, although being trained on only two objects at a time. This is possible by encoding the objects of the scene in images as input to the neural network, instead of a fixed feature vector. Results show runtime improvements of several magnitudes. Video: https://youtu.be/i8yyEbbvoEk

I. INTRODUCTION

TAMP combines symbolic action-sequence reasoning with continuous motion planning, but combinatorial infeasibility makes conventional search expensive. The paper addresses this by predicting goal-conditioned action sequences from scene images, while targeting generalization across varying object counts.

Motivation: TAMP combines symbolic reasoning with continuous motion planning, but many infeasible action sequences require expensive motion-planning evaluations.Infeasibility often arises from kinematic or geometric constraints, and detecting it can take longer than finding a feasible motion.
Generalization: Encoding objects and goals in image space enables generalization to scenes with many and varying numbers of objects despite training on only two objects at a time.The representation avoids requiring a fixed-dimensional object feature vector.
Approach: Goal-conditioned prediction can directly generate sequences rather than searching over many candidates, ideally reducing solution to one motion-planning problem.This contrasts with approaches that predict feasibility only for individual actions.
Approach: The proposed convolutional recurrent network predicts promising action sequences from an initial scene image and task goal to guide TAMP.The predicted sequences parameterize a nonlinear trajectory-optimization problem.
Integration: The method also integrates the predictor into the underlying TAMP tree-search algorithm.This provides a learned mechanism for guiding discrete search.

II. RELATED WORK

Prior work learns representations, heuristics, or single-action feasibility models for planning, but these approaches generally retain discrete search over action sequences. This work instead generates goal-conditioned sequences from images while targeting multi-object generalization.

Learned representations: Related research also learns compact scene representations for planning or reinforcement learning in latent spaces.These approaches differ from the paper’s recurrent reasoning about long-term action effects without direct state prediction.
Learning heuristics: Existing learning-based TAMP methods often guide discrete search heuristically, so many motion-planning problems may still need to be solved.This limitation motivates directly generating action sequences.
Scene representations: Prior scene encodings include fixed object-specific vectors, convolutional or graph representations, and rendered simulator images.Fixed vectors can assume a constant object set, whereas image-based encodings support varying scenes.
Single-action predictors: The closest related methods predict feasibility for a single action, leaving their use for action-sequence prediction unclear.The paper distinguishes sequence-level combinatorial complexity from single-step feasibility.
This work: The paper presents learning to generate action sequences for optimization-based TAMP from an initial scene image and goal, with generalization to multiple objects.This positions the method as sequence-level rather than single-action prediction.

III. LOGIC GEOMETRIC PROGRAMMING FOR TASK AND MOTION PLANNING

Logic Geometric Programming combines a discrete symbolic decision process with nonlinear trajectory optimization over continuous configurations. Its tree search assigns optimization problems to symbolic action sequences and accepts a solution when a goal-state sequence is feasible.

LGP formulation: LGP parameterizes nonlinear trajectory optimization with a discrete symbolic variable whose transitions are constrained by first-order logic.The continuous path and its constraints depend on the symbolic state sequence.
Tree search: Solving LGP requires searching a decision tree in which each node represents a nonlinear trajectory-optimization program.A feasible program at a symbolic goal leaf yields a TAMP solution.
State and feasibility: The initial continuous and symbolic states depend on the scene, while action applicability and successor states determine the symbolic sequence.A feasible action sequence is defined by the existence of a path satisfying the associated constraints.
Continuous path: The optimization path is globally continuous across K phases, with smoothness required within each fixed-duration phase.The number of phases is itself part of the decision problem.

A. Multi-Bound LGP Tree Search and Lower Bounds

The method guides LGP tree search by predicting promising action sequences, reducing reliance on expensive nonlinear programs while preserving solution completeness.

High combinatorial complexity makes LGP search expensive because many early-feasible action choices later become infeasible, requiring numerous nonlinear programs.
The ideal sequence would reach the symbolic goal while requiring only one nonlinear program, yielding a significant LGP speedup.
The network is integrated as a heuristic so search remains able to find a solution even when predictions are imperfect.
The predictor scores actions conditioned on the scene, goal, and previous decisions, selecting actions with the highest probability of leading to a feasible complete sequence.

B. Training Targets

Training converts sampled feasible and infeasible action sequences into binary prefix labels indicating whether each decision remains promising.

Training data consists of sampled scenes, goals, action sequences, and feasibility outcomes for the resulting nonlinear programs.
Feasible sequences receive positive labels for every prefix, directing the network toward decisions that remain on a feasible solution path.
An infeasible sequence receives a positive prefix label when another feasible training sequence shares that prefix, otherwise the label is zero.
The transformed labels enable supervised sequence learning with weighted binary cross-entropy and produce a more balanced training dataset.

C. Input to the Neural Network – Encoding a, g and S

The input representation separates fixed action and goal symbols from scene-dependent object references, allowing the network to handle varying object counts.

Grounded actions and goals combine an operator symbol with the objects they reference, so object references must encode geometric scene information.
Because operator-symbol sets are fixed while object counts vary, symbols can use fixed one-hot encodings and objects can use a variable-compatible representation.
The action-object-image mapping encodes a scene and referenced object tuple as a multichannel image suitable for neural-network input.

2) Encoding the objects O and Og in the image space:

Objects and goals are encoded as image channels and processed by a convolutional recurrent network that predicts action probabilities during tree search.

2) Encoding the objects O and Og in the image space:: Each action-object image includes object-focused masks plus the complete scene, preserving both attention to referenced objects and their geometric relations to others.
2) Encoding the objects O and Og in the image space:: Action-object and goal-object images are encoded by a CNN, while discrete action and goal symbols use one-hot inputs to fully connected layers.
3) Network Architecture:: The recurrent network stores history through a hidden state because it observes only the initial scene configuration while predicting successive actions.
E. Algorithm: During tree search, predictions guide node expansion and nonlinear optimization is attempted only for sufficiently promising goal-reaching leaves.
E. Algorithm: The search remains complete: if a feasible sequence of length at most Kmax exists, prediction errors do not prevent finding it.
E. Algorithm: Caching encodings and hidden states means each search step queries only the recurrent, smaller network component.

F. Alternative: Recurrent Feasibility Classifier

The recurrent feasibility classifier extends single-action feasibility prediction to action sequences, but feasibility alone does not ensure long-term task completion.

F. Alternative: Recurrent Feasibility Classifier: The recurrent classifier evaluates the feasibility of an action sequence independently of whether it reaches a symbolic goal.This extends prior single-action feasibility classification to sequence-level prediction within the TAMP tree search.
F. Alternative: Recurrent Feasibility Classifier: The alternative is integrated into the TAMP search procedure to guide evaluation of candidate sequences.The supplied algorithm passages show expansion, leaf collection, threshold adjustment, and selection by prediction.
F. Alternative: Recurrent Feasibility Classifier: Fig. 3 depicts four integer assignments for the grasp operator.These assignments correspond to the four discrete grasping choices described for one robot arm.
F. Alternative: Recurrent Feasibility Classifier: A feasible action prefix may still fail to solve the TAMP problem over the remaining planning horizon.Therefore, local feasibility does not guarantee that the full sequence will reach the task goal.

V. EXPERIMENTS

The experiments use a tabletop manipulation scenario with two robot arms and box-shaped objects, where the goal is to move objects to target locations.

V. EXPERIMENTS: The evaluation considers two Franka Emika Panda arms manipulating multiple box-shaped objects on a tabletop.A typical scene contains objects and red-square target locations.
V. EXPERIMENTS: The task is to move an object to different target locations represented by red squares.The planning domain includes grasp and place operators, with grasp variants corresponding to different box surfaces.
V. EXPERIMENTS: Grasp actions select a robot arm, one of four integer grasp assignments, and an object, while geometric constraints enforce valid alignment and placement.The grasp operator combines symbolic choices with equality and inequality constraints on the gripper configuration.

1) Action Operators and Optimization Objectives:

The scene properties create substantial combinatorial difficulty because occupancy and reachability determine which action sequences can produce feasible manipulation plans.

2) Properties of the Scene:: Target occupancy and robot-arm reachability make the manipulation task challenging for TAMP algorithms.Plans may require choosing different arms across phases and coordinating both arms.
2) Properties of the Scene:: The number of action sequences reaching a symbolic goal grows with both sequence length and the number of objects.Table I presents these sequences as candidates for feasible solutions, illustrating the task’s combinatorial complexity.
2) Properties of the Scene:: The method avoids handcrafted size-based simplifications and selects promising sequences from the scene’s relational geometry.For example, whether an obstructing object must first be removed depends on the relative geometry and sizes involved.
2) Properties of the Scene:: The dataset contains 30,000 scenes with two objects, sampled across object sizes, positions, orientations, and target locations.Half the scenes place a non-goal object directly on the target to include occupied-target cases.
2) Properties of the Scene:: The network uses ADAM with learning rate 0.0005 and batch size 48, while oversampling feasible sequences to address dataset imbalance.At least 16 samples per batch are drawn from feasible sequences.

C. Performance – Results on Test Scenarios

On test scenarios, the neural network often produces feasible sequences immediately and substantially outperforms LGP tree search, while retaining its advantage over a recurrent classifier as sequence lengths increase.

C. Performance – Results on Test Scenarios: For sequence lengths 2 and 3, the first predicted sequence is feasible, requiring no search and only one NLP; at length 6, median runtime is about 2.3 s.For lengths 5 and 6, at least half the cases require fewer than two NLPs, and upper runtime whiskers remain below 7 s.
C. Performance – Results on Test Scenarios: The neural network is 46 times faster at sequence length 4, 100 times faster at length 5, and 705 times faster at length 6 than LGP tree search.LGP tree search times out on 132 of 2,705 test cases, compared with 3 cases for the neural network.
C. Performance – Results on Test Scenarios: The goal-conditioned network achieves higher speedups than the recurrent classifier, with the advantage remaining relatively constant as sequence length increases.The classifier fails to find 22 solutions, compared with 3 for the proposed approach, and its query time becomes visible.
C. Performance – Results on Test Scenarios: A model trained only with two objects is evaluated on fixed test scenes containing one or more objects.This experiment tests generalization to varying object counts without constructing a dataset for every combination.

G. Generalization to Cylinders

The network was tested on cylinder-containing scenes despite training only on box-shaped objects, with performance largely preserved across action-sequence lengths. The broader framework also produced feasible solutions efficiently and handled additional objects without confusing predictions.

G. Generalization to Cylinders: The cylinder evaluation used 200 scenes containing two cylinders, three cylinders, or mixed box-and-cylinder configurations with varied sizes, positions, orientations, and targets.Cylinders with larger radii were included to create appearances clearly different from the training data.
G. Generalization to Cylinders: Except for action sequences of length 6, cylinder scenes showed no performance drop compared with box-shaped scenes.Even for length 6, runtimes remained very low compared with LGP tree search.
G. Generalization to Cylinders: In a real-world scene, the network proposed a handover solution for an unreachable target object, while an additional object did not confuse its predictions.The planned trajectories were executed open-loop.
G. Generalization to Cylinders: The image-based representation enabled generalization beyond the two-object, box-shaped training setting to multiple objects and other shapes to some extent.The paper identifies this learned representation as supporting high performance despite the broader test conditions.
G. Generalization to Cylinders: The method's main limitation is its assumption that the initial scene image contains sufficient information, excluding total occlusions and other ambiguities.