Source-linked AI summary
Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration
De-An Huang, Suraj Nair, Danfei Xu, Yuke Zhu, Animesh Garg, Li Fei-Fei, Silvio Savarese, Juan Carlos Niebles
TL;DR
The paper addresses how to execute an unseen complex task from a single video demonstration. It proposes Neural Task Graph Networks with a Conjugate Task Graph intermediate representation, and reports generalization across simulated and real-world task data without dense hierarchical supervision.
Problem
One-shot visual imitation must generalize from a single video demonstration to unseen, long-horizon tasks despite difficult visual learning and task complexity.
Method
NTG modularizes the demonstration and derived policy with a graph generator and execution engine based on a Conjugate Task Graph.
Results
NTG outperforms unstructured and hand-designed hierarchical methods across simulated tasks and real-world data, while requiring less detailed supervision.
Takeaways & Limitations
Explicit compositionality supports one-shot visual imitation and generalization to unseen tasks across diverse task domains.
Takeaways & Limitations
Without a surgical environment, the policy cannot be directly evaluated on JIGSAWS and is assessed by task-graph prediction instead.
Abstract
from arXiv · showhide
Our goal is to generate a policy to complete an unseen task given just a single video demonstration of the task in a given domain. We hypothesize that to successfully generalize to unseen complex tasks from a single video demonstration, it is necessary to explicitly incorporate the compositional structure of the tasks into the model. To this end, we propose Neural Task Graph (NTG) Networks, which use conjugate task graph as the intermediate representation to modularize both the video demonstration and the derived policy. We empirically show NTG achieves inter-task generalization on two complex tasks: Block Stacking in BulletPhysics and Object Collection in AI2-THOR. NTG improves data efficiency with visual input as well as achieve strong generalization without the need for dense hierarchical supervision. We further show that similar performance trends hold when applied to real-world data. We show that NTG can effectively predict task structure on the JIGSAWS surgical dataset and generalize to unseen tasks.
1. Introduction
The paper targets one-shot visual imitation for complex unseen tasks, arguing that compositional task structure enables more data-efficient generalization from a single video demonstration.
- Motivation: The work learns policies for unseen tasks from single video demonstrations after training on seen in-domain tasks.The setting directly uses visual videos rather than annotated state trajectories.
- Motivation: Long-horizon visual imitation remains difficult because visual learning and task complexity increase the demand for data efficiency.Cooking and assembly are cited as examples of inherently long-horizon, hierarchical real-world tasks.
- Approach: NTG modularizes the demonstration and policy through a task-graph intermediate representation generated from the video and executed in the environment.The framework contains a graph generator and an execution engine.
- Approach: The Conjugate Task Graph represents actions as nodes and states as edges, avoiding explicit decomposition over an unbounded visual state space.This action-centered representation also supports alternate action sequences under stochastic dynamics.
- Results: NTG improves data efficiency and outperforms unstructured and strongly supervised hierarchical methods across simulated tasks and real-world surgical data.The reported evaluation covers Block Stacking, Object Collection, and JIGSAWS surgical videos without requiring detailed hierarchical supervision.
- Contributions: The paper’s contributions are compositional task and policy representations, NTG, and CTG-based handling of novel visual-state decomposition.
2. Related Work
Prior work studies imitation, task planning, structural video understanding, and compositional models, while NTG combines these directions for one-shot visual imitation of complex tasks.
- One-shot Visual Imitation: The one-shot visual imitation setting trains on seen tasks and applies the learned model to a single demonstration from an unseen task.This setting is illustrated by the transition from Tasks 1 and 2 to unseen Task 3.
- Imitation Learning: One-shot imitation methods translate a single demonstration into an executable policy, but the related NTP method requires strong hierarchical frame labels.NTP also shows a noticeable performance drop with visual state, whereas NTG uses only the demonstration action sequence during training.
- Task Planning and Representations: Task-planning research represents high-level plans, low-level states, hierarchical subtasks, task-motion relations, or behavior-based systems.
- Structural Video Understanding: Structural video understanding often uses language, scripts, or captions as supervision because video annotation is difficult.NTG instead focuses on how structure supports task learning and assumes annotations for seen tasks.
- Compositional Models in Vision and Robotics: Compositionality has improved generalization in visual question answering and policy learning, motivating its use for data-efficient visual imitation of complex tasks.
3. Problem Formulation
The problem is to learn a model that converts one video demonstration of an unseen task into a policy conditioned on visual observations.
- Problem Setup: A demonstration is a video sequence of visual observations that completes a task, while tasks are divided into seen training tasks and unseen evaluation tasks.
- Learning Objective: The learned model φ maps a demonstration d to a policy πd(a|o) that should complete the demonstrated novel task from visual observation o.Training uses demonstrations and potentially extra supervision from Tseen; testing supplies only an unseen-task demonstration.
4. Neural Task Graph Networks
NTG represents unseen-task demonstrations with a Conjugate Task Graph and derives a policy through graph-based execution. Its generator infers graph edges from action order and graph completion, while execution localizes nodes and classifies next-action transitions from visual observations.
- 4. Neural Task Graph Networks: NTG decomposes policy learning into a graph generator that builds a task graph from one demonstration and an execution engine that acts from the graph.The task graph provides compositional structure for both the demonstration and the derived policy.
- 4.1. Neural Task Graph Generator: The generator uses a Conjugate Task Graph whose nodes are observed actions, inferring edges from demonstrated order and completing unobserved interchangeable transitions.Training assumes domain actions are observed in seen tasks; graph completion adds edges compatible with initialized transitions.
- 4.2. Neural Task Graph Execution: The execution engine localizes the current action node from visual input and classifies the next edge using latent action preconditions, producing a reactive policy.The edge classifier selects the next action, while node embeddings support localization and graph-structured transition prediction.
- 4.2. Neural Task Graph Execution: Repeated visits to the same action node can produce different next actions because edge classification uses the current visual state and learned transition structure.In the illustrated execution, the second visit to Move B correctly leads to Place D.
- 4.3. Learning NTG Networks: NTG training requires raw visual observations and flat action sequences rather than state-action pairs or manually defined hierarchical supervision.Demonstrations within a task are aggregated to supervise model components, and the action sequences define paths used for graph learning.
5. Experiments
NTG is evaluated for one-shot visual imitation across simulated, ablation, object-collection, and surgical settings. Results indicate that compositional task structure supports generalization to unseen configurations, execution orders, object counts, and surgical task behavior.
- 5.2. Ablation Analysis of NTG Model Components: GCN enables solving an unseen sorting task in an execution order different from the demonstration, while Edge Classifier is required to select actions from newly introduced edges.The ablation identifies Demo Interpreter and Node Localizer as almost indispensable, and both GCN and Edge Classifier as necessary for alternate execution orders.
- 5.1. Evaluating Block Stacking in BulletPhysics: With more than 100 training tasks, NTG significantly outperforms baselines on unseen Block Stacking configurations, while raw visual NTG matches NTP using full state.With only 50 tasks, methods with extra hierarchical or detection supervision can outperform NTG.
- 5.3. Evaluating Object Collection in AI2-THOR: NTG significantly outperforms the Flat Policy when trained on 2 and 4 objects and tested on 1, 3, and 5 objects in Object Collection.The evaluation also varies visual appearances and object locations, requiring the model to infer task state rather than match demonstration appearance.
- 5.4. Evaluating Real-world Surgical Data: The full NTG model substantially improves NLL over the no-graph and uniform-policy variants and is most consistent with expert demonstrations on JIGSAWS.Lower NLL corresponds to better explanation of demonstrations and better capture of task structure.
- 5.4. Evaluating Real-world Surgical Data: On unseen Needle Passing tasks, NTG predicts recovery edges that return to Orienting Needle after failures, matching the described ground-truth behavior.The model is trained on Knot Tying and Suturing and evaluated on Needle Passing.
6. Conclusion
NTG incorporates task compositionality into both the intermediate representation and policy, with CTG generation and execution supporting unseen visual states. It outperforms unstructured and hand-designed hierarchical methods across simulated and real-world data.
- NTG outperforms methods with unstructured representations and hand-designed hierarchical structures across diverse tasks.The evaluation includes a photo-realistically rendered simulated environment and a real-world dataset.