PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Energy Fields for Inference-Time Procedure Planning in Instructional Videos

Mohamed Afham, Christoph Reich, Oliver Hahn, Daniel Cremers, Stefan Roth

arXiv:2608.16457v1cs.CVcs.AI

TL;DR

Procedure planning methods often predict action sequences without explicitly filtering irrelevant actions during inference. CEFITO instead learns an action-conditioned energy field and performs task-constrained optimization, achieving state-of-the-art accuracy on two instructional-video benchmarks.

  • Problem

    Existing procedure planners lack inference-time mechanisms to disregard irrelevant actions while selecting goal-directed action sequences.

  • Method

    CEFITO learns an action-conditioned energy field with contrastive learning and optimizes action sequences at inference time under task constraints.

  • Results

    CEFITO achieves state-of-the-art accuracy on CrossTask and COIN, improving CrossTask SR over SCHEMA by 2.38% at T = 3.

  • Takeaways & Limitations

    Inference-time optimization over task-relevant actions provides an effective procedure-planning framework without relying on large language models.

  • Takeaways & Limitations

    For planning horizon T = 6, CEFITO is two orders of magnitude slower, motivating approximate optimization strategies to reduce inference runtime.

Abstract

from arXiv · show

Procedure planning seeks to estimate a sequence of actions to transition from an observed initial state to a given goal state. Current procedure planning approaches directly predict action sequences from latent representations using feed-forward neural networks or diffusion-based inference. These paradigms treat every action as plausible, lacking the ability to enforce task-specific logical constraints that render certain actions irrelevant or not plausible. We propose CEFITO, a procedure planning approach that learns a predictor to express an action-conditioned representation space. Based on this representation space, we formulate procedure planning as a task-constrained optimization problem. Unlike prior methods, CEFITO explicitly reasons over the action space by omitting irrelevant actions during inference-time planning. This reformulation enables effective procedure planning and achieves state-of-the-art accuracy on two established procedure planning benchmarks.

1 Introduction

CEFITO reframes procedure planning as inference-time energy minimization over an action-conditioned representation, addressing the inability of feed-forward and diffusion approaches to omit irrelevant actions. It learns this representation contrastively and searches a restricted, task-relevant action subset without requiring intermediate-state supervision.

  • Motivation: Procedure planning in instructional videos estimates an action sequence from a visual initial state to a desired goal state.Planning supports reaching desired goals across real-world tasks, including robotic navigation, autonomous driving, virtual reality, and healthcare.
  • Limitations of Existing Methods: Existing approaches map latent representations to action-sequence distributions using feed-forward or diffusion-based inference, limiting their ability to disregard irrelevant actions.These approaches include transformers, diffusion models, and task-specific procedural knowledge graphs, whereas human planning iteratively considers possible actions and their consequences.
  • CEFITO: CEFITO learns an action-conditioned representation space, or energy field, with contrastive learning and uses it for inference-time optimization.The representation maps action sequences from an initial observation toward a predicted goal latent, whose distance to the encoded goal latent defines candidate energy.
  • Contributions: CEFITO formulates procedure planning as energy minimization and restricts inference-time search to a task-relevant action subset without intermediate-state supervision.Its contributions include contrastive energy learning, task-constrained search, and evaluation on established instructional-video procedure-planning benchmarks.

2 Related Work

Related work frames procedure planning as visual state-to-goal action sequencing, connects it to inference-time optimization in robotics, and situates CEFITO within contrastive representation learning. CEFITO reformulates planning as inference-time optimization over an action-conditioned energy field.

  • Procedure Planning in Instructional Videos: Procedure planning in instructional videos estimates action sequences that transition from an initial visual observation to a desired visual goal state.Both states are provided as video frames, and the task is closely related to general task planning.
  • Procedure Planning in Instructional Videos: Early procedure-planning approaches used sequential latent-space modeling or adversarial policy learning.
  • Inference-Time Optimization: Inference-time optimization is established in robotics control and planning, where action sequences are optimized online using forward predictive models.Visual foresight methods combined pixel-level rollouts or learned dynamics with model predictive control to plan trajectories during inference.
  • Inference-Time Optimization: CEFITO reformulates procedure planning as an inference-time optimization problem using an action-conditioned energy field.
  • Contrastive Learning: Contrastive learning learns expressive representations by distinguishing positive and negative data points and supports tasks such as action recognition, anticipation, and temporal localization.

3 Method: CEFITO

CEFITO reformulates procedure planning as inference-time optimization over a learned action-conditioned energy field. It combines contrastive predictor training with task-specific search that excludes irrelevant actions.

  • Energy-field formulation: CEFITO maps an initial state and candidate action sequence to a predicted goal embedding, then uses distance to the observed goal embedding as an energy function.Planning selects the sequence minimizing d(Pθ(xs, a1:T), xg).
  • Contrastive predictor learning: The predictor Pθ is trained with a margin-based triplet loss, bringing correct-sequence predictions toward the goal while separating wrong-sequence predictions.Per-negative margins τi enforce minimum separation and prevent collapsed embeddings.
  • Negative sampling: Mixed-negative sampling combines hard negatives that alter action ordering or composition with easy negatives drawn from other tasks to model high- and low-level structure.The hard/easy balance is controlled by r, with rN hard and (1−r)N easy negatives.
  • Adaptive margins: Adaptive margins assign each negative a margin based on action overlap, distinguishing near-miss sequences from unrelated action sequences.Margins are bounded by τi ∈[τmin, τmax].
  • Task-constrained inference: At inference, a task classifier predicts the high-level task and restricts discrete search from the full action space to the task-specific subspace A(ˆc), omitting irrelevant actions.The classifier takes (xs, xg) and predicts ˆc using cross-entropy training.

4 Experiments

CEFITO achieves state-of-the-art procedure-planning accuracy on CrossTask and COIN, outperforming recent methods across multiple horizons and metrics. Ablations identify contrastive learning and pretrained text encoders as important components, while classifier errors and limited data constrain performance.

  • Datasets and Metrics: CEFITO is evaluated on CrossTask and COIN using five random seeds and mean performance with 90% confidence intervals.CrossTask contains 2,750 videos, 18 tasks, and 105 actions; COIN contains 11,827 videos, 180 tasks, and 778 actions.
  • CrossTask Results: CEFITO outperforms recent state-of-the-art methods on CrossTask for both T = 3 and T = 4 across all procedure-planning accuracy metrics, including LLM-based SCHEMA.The reported example exceeds SCHEMA by 2.38%.
  • COIN Results: CEFITO achieves the best mean SR, mAcc, and mIoU at T = 3 and the best SR and mAcc at T = 4 on COIN.Improvements over ViterbiPlanNet are 0.12% SR at T = 3 and 0.33% SR at T = 4.
  • Component Analysis: +23.93 SR at T = 3 results from adding the triplet contrastive loss to the L2-regression baseline.Contrastive learning enforces an energy field that places correct predicted goals near xg and incorrect ones far from xg.
  • Text Encoder Analysis: Both pretrained text encoders outperform random embeddings across all metrics, and CLIP consistently outperforms Flan-T5-base.The paper attributes CLIP’s advantage to multimodal contrastive pretraining that aligns text with visual concepts.
  • Limitations: About 20% of COIN validation samples contain wrong action-sequence predictions caused by the task classifier, while CEFITO performs suboptimally on the much smaller NIV dataset.The paper links NIV’s failure to its roughly two-orders-of-magnitude smaller video count and possible contrastive-learning collapse.

5 Conclusion

CEFITO is an inference-time optimization framework for procedure planning in instructional videos. It learns to predict goal embeddings from candidate action sequences and initial states, achieving state-of-the-art accuracy on CrossTask and COIN without relying on large language models.

  • Framework: CEFITO introduces inference-time optimization for procedure planning in instructional videos.It formulates planning as task-constrained inference-time optimization.
  • Predictor model: CEFITO maps a candidate action sequence and initial state to a predicted goal embedding.This predictor model enables task-constrained inference-time optimization for planning.
  • Results: CEFITO achieves state-of-the-art accuracy on CrossTask and COIN.The conclusion reports these results across both established procedure-planning benchmarks.
  • Planning approach: CEFITO performs inference-time planning without relying on large language models.This distinguishes its optimization-based planning approach from current approaches described in the conclusion.

Supplementary Material · A Auxiliary Action Sequence Reconstruction

The supplementary material adds implementation, baseline, and qualitative-result details, while Section A describes an auxiliary action-sequence reconstruction loss that enriches intermediate representations. The pipeline uses a shared predictor to reconstruct ground-truth actions from predicted latent states.

  • Supplementary Material: The supplement provides additional details on auxiliary reconstruction, implementation, baseline methods, and qualitative results.These materials are organized into Sections A–D.
  • A Auxiliary Action Sequence Reconstruction: The auxiliary loss Laux complements contrastive loss Lc by supervising action reconstruction to obtain expressive intermediate representations.Lc supervises the predicted goal state, whereas Laux compares reconstructed and ground-truth action sequences.
  • A Auxiliary Action Sequence Reconstruction: The video encoder maps initial and goal observations vs and vg to latent states xs and xg.The text encoder maps the ground-truth action sequence to action tokens a1:T.
  • A Auxiliary Action Sequence Reconstruction: In the first predictor pass, the encoded initial state, action tokens, and goal state produce predicted intermediate states and a predicted goal state.The predictor Pθ is applied to generate the latent-state sequence used by the reconstruction pipeline.
  • A Auxiliary Action Sequence Reconstruction: The second pass feeds the initial state, predicted intermediate latents, goal-state token, and learnable action mask tokens into the predictor.This pass uses the same predictor with shared weights.
  • A Auxiliary Action Sequence Reconstruction: The weight-shared predictor reconstructs predicted action tokens ˜a1:T from the latent states and mask tokens.The predicted tokens are then mapped back into the action space.
  • A Auxiliary Action Sequence Reconstruction: The auxiliary loss penalizes discrepancy between reconstructed actions ˜a1:T and ground-truth actions a1:T.The complete auxiliary action-sequence reconstruction pipeline is visualized in Fig. A.1.

B Implementation Details · Architecture.

CEFITO implements its predictor as a compact causal transformer conditioned on initial and goal visual states plus tokenized actions. Visual inputs are extracted with a pretrained S3D encoder using temporal windows around the observed states.

  • Architecture.: The predictor Pθ is implemented as a causal masked transformer.This architecture enforces autoregressive conditioning during prediction.
  • Architecture.: The transformer uses four blocks, each with six attention heads.
  • Architecture.: The model has a hidden dimension of 384.
  • Architecture.: Pθ contains 8.5 M learnable parameters.
  • Architecture.: Pθ receives visual features from the initial state xs and goal state xg.
  • Architecture.: Pθ also receives tokenized actions as input.
  • Architecture.: Visual features are extracted with the S3D encoder pretrained on HowTo100M.
  • Architecture.: The visual encoder uses a temporal window of 3-frames around vs and vg.

Training. · C Baselines

Training uses AdamW, hard/easy negative sampling, margin settings, early stopping, and a separately trained task classifier, with an auxiliary reconstruction loss supervising predictor outputs. The baselines cover diffusion-based distribution fitting, procedural knowledge graphs, structured state changes, LLM-enhanced planning, and differentiable Viterbi decoding.

  • Training.: The predictor weights θ are trained with AdamW using weight decay 1 × 10−3 and learning rate 5 × 10−4.
  • Training.: An auxiliary loss trains shared-weight predictor applications to reconstruct intermediate and goal latents, then masked action tokens from the initial and goal states.The reconstructed action tokens are supervised by the ground-truth action sequence a1:4.
  • Training.: The predictor is trained for 200 epochs with early stopping, while the task classifier uses learning rate 1 × 10−4 and no weight decay.
  • C Baselines: PDPP formulates procedure planning as diffusion-based distribution fitting conditioned on observations, initial state, and goal state, producing uncertain action-sequence samples at inference.
  • C Baselines: KEPP extends PDPP with a probabilistic procedure knowledge graph built from training data through iterative edge addition and reweighting, providing a retrieval signal for diffusion inference.
  • C Baselines: SCHEMA represents each procedure step as a state change, uses an LLM for language descriptions, and aligns them with visual observations through cross-modal contrastive learning.PlanLLM builds on these state descriptions but generates free-form planning output with LLM-enhanced step decoding and mutual-information maximization linking text with visual states.
  • C Baselines: ViterbiPlanNet incorporates procedural knowledge into end-to-end training through a differentiable Viterbi layer that embeds a procedural knowledge graph using smooth relaxations.This approach differs from KEPP by integrating the knowledge graph into Viterbi decoding during training.

D Qualitative Results

Qualitative examples across eight CrossTask tasks show that CEFITO identifies the ground-truth procedure as its top-ranked plan while retaining plausible alternatives among the top five. These alternatives are incorrect because they can involve step reordering or action repetition.

  • Qualitative Results: Across 8 CrossTask tasks, CEFITO’s top-ranked candidate plan matches the ground-truth action sequence.The top-ranked plan is defined as the candidate with the lowest L2 distance to xg.
  • Qualitative Results: All top-5 candidates are plausible plans, although some are incorrect alternatives.The passage describes these alternatives as plausible despite not matching the ground-truth sequence.
  • Qualitative Results: Incorrect alternatives can involve step reordering, such as “Flip pancake” versus “Take pancake from pan” in “Make pancakes.”The example illustrates a reordered action sequence in the Make pancakes task.
  • Qualitative Results: Incorrect alternatives can also involve action repetition, such as “Flip steak” in “Grill steak.”The passage identifies repeated actions as another source of incorrect alternatives.

Failure cases.

CEFITO can rank the correct action sequence below first place while retaining it among the top-5 candidates, but some failures place the ground truth outside the top-5. The reported failure cases occur on CrossTask and include examples where the model fails to infer the correct sequence within its top-5 candidates.

  • Failure cases.: On CrossTask, some correct action sequences are not top-ranked but remain within the top-5 candidates.Examples include “Grill steak”, “Make pancakes”, and “Make meringue”.
  • Failure cases.: In other examples, CEFITO fails to infer the correct action sequence within the top-5 candidates.The failure cases are shown in Figs. A.4 and A.5.
  • Failure cases.: Some failure cases have the ground-truth sequence outside the top-5 candidates.The figures mark cases where the top-ranked candidate plan does not match the ground truth.
Loading 2608.16457v1…