Source-linked AI summary
Procedure Planning in Instructional Videos
Chien-Yi Chang, De-An Huang, Danfei Xu, Ehsan Adeli, Li Fei-Fei, Juan Carlos Niebles
TL;DR
Procedure planning asks which high-level actions can move a current visual observation toward a goal in instructional videos. DDN learns plannable latent representations through conjugate state-action relationships, and the paper reports significantly better planning than existing methods.
Problem
Procedure planning asks what high-level actions can move a current visual observation toward a visual goal in unstructured instructional videos, a step toward autonomous agents handling complex everyday tasks.
Source: §1 · pp. 1–2Method
Dual Dynamics Networks (DDN) learn plannable latent representations by jointly modeling forward state transitions and conjugate relationships between states and actions.
Source: §1 · p. 2Results
DDN learns plannable representations and significantly outperforms a variety of baselines across different metrics for procedure planning in real-world instructional videos.
Source: §1 · pp. 2–3, §5 · p. 14Takeaways & Limitations
The work is a step toward enabling agents to learn from real-world demonstrations and plan for complex tasks like humans.
Source: §5 · p. 14The planning formulation assumes latent-space forward dynamics that predict future states from current states and applicable actions.
Source: §3.1 · p. 4
Abstract
from arXiv · showhide
In this paper, we study the problem of procedure planning in instructional videos, which can be seen as a step towards enabling autonomous agents to plan for complex tasks in everyday settings such as cooking. Given the current visual observation of the world and a visual goal, we ask the question "What actions need to be taken in order to achieve the goal?". The key technical challenge is to learn structured and plannable state and action spaces directly from unstructured videos. We address this challenge by proposing Dual Dynamics Networks (DDN), a framework that explicitly leverages the structured priors imposed by the conjugate relationships between states and actions in a learned plannable latent space. We evaluate our method on real-world instructional videos. Our experiments show that DDN learns plannable representations that lead to better planning performance compared to existing planning approaches and neural network policies.
1 Introduction
The introduction frames goal-conditioned planning in instructional videos as a step toward autonomous performance of complex everyday tasks, while identifying the learning of plannable representations from unstructured videos as the key technical challenge.
1 Introduction
- Procedure planning asks for a sequence of high-level actions that can move the underlying state of a current visual observation toward the state of a visual goal.The paper presents this as a step toward enabling autonomous agents to plan for complex everyday tasks.Source: §1 · pp. 1–2
- The central technical challenge is learning structured, plannable state and action spaces from unstructured videos, whose complex scenes and abstract tasks resist predefined symbolic planning.Classical symbolic approaches require a given set of predicates for a well-defined state space.Source: §1 · p. 2
- Dual Dynamics Networks (DDN) impose structured priors on latent representations by jointly modeling forward state transitions and conjugate state-action relationships.The method treats states as preconditions for subsequent actions as well as modeling actions as transformations between states.Source: §1 · p. 2
- On real-world instructional videos, DDN significantly outperforms existing planning baselines, generalizes to variations in start and goal observations, and outperforms existing methods on walkthrough planning.The introduction also reports significantly better results than planning and video-understanding methods for procedure planning.Source: §1 · pp. 2–3
2 Related Work
Related work covers task planning with predefined domains, planning from pixels, and instructional-video understanding; this paper instead emphasizes goal-conditioned actions in complex visual settings.
Task Planning
- Classical task planning generally assumes a predefined planning domain, whereas this work plans over visual and semantic representations without a hand-defined symbolic domain.Related work describes task planning as finding task-level actions that reach a desired goal from the current state.Source: §2 · p. 3
Planning from Pixels
- Pixel-based planning spans manipulation, navigation, and locomotion, but long-horizon planning from unstructured observations remains challenging.UPN assumes differentiable actions; this work adds conjugate dynamics to a forward-dynamics formulation, while causal InfoGAN uses toy data rather than real-world videos.Source: §2 · p. 3
Understanding Instructional Videos
- Instructional-video research studies recognition, segmentation, state understanding, summarization, retrieval, prediction, and related tasks, while this paper conditions future actions on both history and goal.The goal-conditioned formulation resolves ambiguity when history alone does not adequately constrain future actions.Source: §2 · pp. 3–4
3 Method
DDN frames procedure planning as latent-space search from a current visual observation to a visual goal, learning state and action representations with forward and conjugate dynamics for procedure and walkthrough planning.
3.1 Problem Formulation
- Given current and goal observations, the model maps them into latent states and searches for a task-level action sequence using forward dynamics.The setup assumes latent forward dynamics that predict the next state from the current state and applicable action.Source: §3.1 · p. 4
3.2 Learning Plannable Representations
- DDN jointly learns forward dynamics T and conjugate dynamics P, using state/action preconditions to constrain latent mappings and applicable actions.This addresses prior approaches’ strong observation–state correspondence assumption and their requirement for continuous, differentiable actions.Source: §3.2 · pp. 5–6
3.3 Planning in Latent Space
- For procedure planning, P samples applicable actions and T predicts successor states, while a bounded priority-queue search selects a plan toward the goal.The search is bounded by a maximum iteration count, goal-distance threshold, and queue size.Source: §3.3 · p. 7, §3.2 · p. 7, Figure 3 · p. 8
- Our model consistently outperforms other baselines as planning horizon T increases; when T ≥ 6, the start observation and visual goal are separated by at least six high-level actions, and all baselines achieve success rates below 1%.Source: Figure 4 · p. 8
- Walkthrough planning finds a length-T path between current and goal observations that maximizes transition scores computed from the learned models.When using clips from one instructional video, this becomes a constrained permutation-path problem.Source: §3.3 · pp. 7–8
4 Experiments
On real-world instructional videos, DDN outperforms baselines in procedure planning and transfers to walkthrough planning, while generalizing to changed start and goal observations.
4 Experiments
- The evaluation uses a fully supervised dataset adapted from 2,750 CrossTask videos, covering 18 tasks and 212 hours of video.The authors leave using unlabeled instructional videos as future work.Source: §4 · p. 9
4.1 Evaluating Procedure Planning
- The full model achieves ∼10% improvement in accuracy and improves success rate by 8 times versus Ours w/o T, and outperforms all baseline approaches on all metrics.Source: Table 1 · p. 10, §4.1 · p. 11
- For unseen start and goal observations, DDN plans essential actions across stages, including four actions for a six-action gap by merging repeats and adding flipping and plating.The example was not seen during training.Source: Figure 6 · p. 12, §4.1 · p. 12
4.2 Evaluating Intermediate States with Walkthrough Planning
- DDN significantly outperforms walkthrough-planning baselines on all metrics; visualizations show correctly ordered intermediate states where baselines miss the tool-fetch step.The challenging tool-fetch step looks visually different from the other tire-changing steps.Source: Table 2 · p. 13, §4.2 · p. 14, Figure 7 · p. 14
5 Conclusion
Dual Dynamics Networks (DDN) learns plannable representations from real-world instructional videos by leveraging structured priors between states and actions. Experiments show it outperforms a variety of baselines across different metrics.
5 Conclusion
- DDN learns plannable representations directly from unstructured real-world instructional videos by leveraging structured priors from state-action relationships.The paper frames this as a step toward enabling autonomous agents to plan for complex tasks like humans.Source: §5 · p. 14
- DDN significantly outperforms a variety of baselines across different metrics in the reported experiments.Source: §5 · p. 14
- Future work will incorporate object-oriented models to further explore object and predicate relations in visually complex environments.Source: §5 · p. 14