Contrastive Energy Fields for Inference-Time Procedure Planning in Instructional Videos

1 Introduction

• Existing approaches map latent representations to action-sequence distributions and cannot disregard irrelevant actions during inference, unlike human iterative planning.The introduction contrasts feed-forward or diffusion-based inference with humans inferring possible actions and optimizing over them.Source: §1 · pp. 1–2
• Modeling action-conditioned visual transitions is challenging because intermediate observations are scarce; existing proxies rely on rigid structures, huge datasets, or large-scale pre-training.Prior proxies include large language models, interpolation, and probabilistic graphs.Source: §1 · p. 2
• CEFITO learns an action-conditioned representation with contrastive learning, then plans by minimizing energy under a task-constrained action search.Its predictor models the transition from an initial state to a goal state given an action sequence; inference omits irrelevant actions.Source: §1 · p. 2, Figure 1 · p. 2
• CEFITO achieves state-of-the-art accuracy on two datasets and different planning horizons without relying on large language models.The paper identifies these as established procedure-planning benchmarks.Source: §1 · p. 3
• The energy-minimization formulation requires no intermediate-state supervision, and inference searches a restricted, task-relevant action subset.The task-constrained search uses the learned energy to plan at inference time.Source: §1 · p. 3

2 Related Work

• Prior procedure planners use sequential or adversarial models, language or task supervision, transformers or diffusion, knowledge graphs, and LLMs.Unlike these approaches, CEFITO learns an action-conditioned representation space with contrastive learning to define an energy function for task-constrained inference.Source: §2 · p. 3

Inference-Time Optimization for Planning.

• Inference-time planning builds on robotics control, visual foresight, latent world models, and visual foundation models [16] [2] [23] [35].Motivated by classical methods, the paper reformulates procedure planning as inference-time optimization using an action-conditioned energy field.Source: Inference-Time Optimization for Planning · pp. 3–4

Contrastive Learning.

• The method uses supervised contrastive learning as a triplet loss to learn an action-conditioned representation for planning at inference time.The related work notes that hard negatives—semantically similar to positive samples—provide informative training signals.Source: Contrastive Learning · p. 4

3.1 Procedure Planning as Inference-Time Optimization

• Given initial and goal video frames, the method extracts latent states with S3D pretrained on HowTo100M, following established procedure-planning approaches.The initial and goal states are encoded as xs and xg, respectively.Source: Problem Formulation · p. 4

Reformulating Procedure Planning.

• CEFITO replaces direct sequence prediction with inference-time search: a predictor maps an action sequence and initial state to a predicted goal embedding, scored by its distance to the observed goal.The action-conditioned distance d(Pθ(xs, a1:T), xg) serves as the energy to minimize.Source: Reformulating Procedure Planning · pp. 4–5, §3.2 · p. 5

3.2 Action-Conditioned Energy Field

• The action-conditioned predictor Pθ is implemented as a transformer, receiving the initial visual latent and CLIP-encoded textual action tokens.It is trained with a margin-based triplet loss to bring predictions for correct sequences toward the observed goal while separating incorrect-sequence predictions.Source: §3.2 · p. 5, Contrastive Training · p. 6

Contrastive Training.

• Mixed negative sampling combines hard sequences from the same task with easy sequences from other tasks, while adaptive margins reflect each negative’s action overlap with the positive sequence.This encourages the energy field to capture both task distinctions and action-order or composition errors; per-negative margins are bounded by τmin and τmax.Source: Contrastive Training · p. 6
• An auxiliary action-reconstruction loss supervises intermediate predictor representations alongside the contrastive loss.The predictor uses intermediate representations and a decoding head to reconstruct the actions, with the full loss L = Lc + λLaux.Source: Contrastive Training · pp. 6–7

3.3 Task-Constrained Optimization for Planning

• At inference, a classifier estimates the high-level task from the initial and goal states, restricting discrete search to a task-specific action subspace and excluding irrelevant actions.The subspace is derived from training data and used to select the candidate sequence whose predicted goal best matches the observed goal.Source: §3.3 · p. 7

Datasets.

• Evaluation uses CrossTask’s 2,750 videos and 105 actions and COIN’s 11,827 videos and 778 actions, with five-seed means and 90 % confidence intervals; SR, mAcc, and element-wise mIoU measure exact matches, stepwise accuracy, and sequence overlap.SR is the strictest metric, and all metrics are reported in percent.Source: Datasets · p. 8, Metrics · p. 8

4.1 Comparison to the State of The Art

• CEFITO outperforms prior approaches on CrossTask across metrics and horizons, including SCHEMA by 2.38 % SR at T = 3 and ViterbiPlanNet by 1.17 % and 0.12 % SR at T = 3 and T = 4; on COIN it leads in mean SR, mAcc, and mIoU at T = 3 and SR and mAcc at T = 4.On COIN, its SR gains over ViterbiPlanNet are 0.12 % at T = 3 and 0.33 % at T = 4; the paper attributes smaller benefits than on CrossTask to fewer videos per task across COIN’s 180-task distribution.Source: Table 2 · p. 9, §4.1 · pp. 9–10

4.2 Analyzing CEFITO

• On CrossTask at T = 3, adding triplet contrastive loss raises SR by +23.93, adaptive margins add +1.16 %, and auxiliary action reconstruction adds +0.88 % over the preceding component configuration.Plain L2 regression to the goal representation severely underperforms ViterbiPlanNet [49]; contrastive learning separates predicted goals for correct and incorrect sequences.Source: Learning Objective · p. 10, Table 3 · p. 10

Text Encoder Analysis.

• Pre-trained text encoders outperform random embeddings across metrics, and CLIP consistently outperforms the larger Flan-T5-base, attributed to multimodal alignment of text with visual concepts.The comparison evaluates random, frozen Flan-T5-base, and frozen CLIP text encoders on CrossTask at T = 3 and T = 4.Source: Text Encoder Analysis · pp. 10–11, Table 4 · p. 10

Negative-Sequence Selection.

• On CrossTask at T = 3, success rate peaks at N = 50 negative samples and r = 0.8, a mixture biased toward hard intra-task negatives while retaining inter-task separation.Performance is low at N = 10, then saturates as N increases further; r = 0 and r = 1 correspond to only easy and only hard negatives, respectively.Source: Negative-Sequence Selection · p. 11, Figure 4 · p. 11

PDPP

• Task-classifier errors constrain inference: accuracy exceeds 90 % on CrossTask, but on COIN the classifier causes wrong action sequences in about 20 % of validation samples; oracle task labels improve both datasets, especially COIN.The oracle replaces the predicted high-level task with ground truth, providing an upper bound and showing that better task classification can improve downstream planning.Source: Task Classifier Analysis · p. 11, Table 5 · p. 11, Table 6 · p. 11
• Inference runtime is manageable and comparable to existing methods for T ≤ 4, but grows exponentially with horizon; CEFITO is about one order slower than PDPP at T = 5 and two orders slower at T = 6.The task constraint reduces the search space, yet larger horizons require many predictor forward passes; beam search and gradient-based inference are suggested as possible accelerations.Source: Limitations & Failed Experiments · p. 12, Figure 5 · p. 12

Limitations & Failed Experiments.

• On NIV, CEFITO achieves suboptimal accuracy; the authors attribute this to the dataset’s 150 videos, versus over 11,000 in COIN, and suspect contrastive learning collapses with so few training videos.The NIV results are reported in Table 7.Source: Limitations & Failed Experiments · p. 13, Table 7 · p. 13

• CEFITO demonstrates state-of-the-art accuracy on the CrossTask and COIN procedure-planning benchmarks.Source: §5 · p. 13
• CEFITO uses task-constrained inference-time optimization, with a predictor mapping candidate action sequences and the initial state to a predicted goal embedding.Source: §5 · p. 13
• Unlike current approaches, CEFITO plans through inference-time optimization without relying on large language models.Source: §5 · p. 13

Supplementary Material

• The supplement also provides further implementation details, summaries of the compared baseline methods, and qualitative results.Source: Supplementary Material · p. 18

A Auxiliary Action Sequence Reconstruction

• The supplement adds an auxiliary action-sequence reconstruction loss to goal-state contrastive supervision to obtain expressive intermediate representations.The contrastive loss supervises only the predicted goal state; the auxiliary loss penalizes reconstructed-versus-ground-truth actions.Source: A Auxiliary Action Sequence… · p. 18
• The video encoder maps the initial and goal observations to latent states, while the text encoder maps the ground-truth action sequence to action tokens.Source: A Auxiliary Action Sequence… · p. 18
• In the first pass, the predictor receives the initial latent, action tokens, and learnable masks for unobserved states to predict intermediate and goal latents.Source: A Auxiliary Action Sequence… · p. 18
• In the second pass, the predictor receives the initial latent, predicted intermediate latents, ground-truth goal latent, and learnable action-mask tokens.Source: A Auxiliary Action Sequence… · p. 18
• The same weight-shared predictor reconstructs action tokens, which are mapped back to the action space and penalized against ground-truth actions.The reconstruction pipeline is shown in Fig. A.1.Source: A Auxiliary Action Sequence… · p. 18

Architecture.

• Pθ is a causal masked transformer with four blocks, six attention heads, hidden dimension 384, and 8.5 M learnable parameters.It receives initial-state and goal-state visual features together with tokenized actions.Source: Architecture · p. 18
• Initial and goal visual features come from frozen S3D pretrained on HowTo100M, using a 3-frame temporal window around each state.Source: Architecture · p. 18
• Before entering Pθ, initial and goal features are normalized along the feature dimension and linearly projected to its hidden dimension.Source: Architecture · p. 18
• Natural-language actions are tokenized and encoded with the pretrained CLIP-ViT-B text encoder, then linearly projected to Pθ’s hidden dimension.CLIP and the visual encoder are frozen; only the linear projections are learned.Source: Architecture · p. 18
• The task classifier is a four-layer multilayer perceptron that takes both initial-state and goal-state features as input.Source: Architecture · p. 18

Training.

• AdamW trains predictor weights θ with weight decay 1 × 10−3 and learning rate 5 × 10−4.Source: Training · p. 18
• Training generates 50 negative samples per positive sequence, sets the hard/easy ratio to 0.8, and uses minimum and maximum margins of 0.01.Source: Training · pp. 18–19
• The predictor trains for 200 epochs with early stopping following KEPP, while the task classifier uses learning rate 1 × 10−4 and no weight decay.Source: Training · p. 19

C Baselines

• PDPP uses conditional diffusion to fit action-sequence distributions from initial and goal states, and KEPP adds a training-data-derived probabilistic procedure knowledge graph as a retrieval signal.Source: C Baselines · p. 19
• SCHEMA aligns LLM-generated state-change descriptions with visual observations, while PlanLLM generates free-form plans and adds LLM-enhanced planning and mutual-information maximization.PlanLLM links commonsense text with visual state; SCHEMA represents each step as a state change.Source: C Baselines · pp. 19–20
• Unlike KEPP, ViterbiPlanNet incorporates a procedural knowledge graph into end-to-end training through a differentiable Viterbi layer with smooth relaxations.Source: C Baselines · pp. 19–20

D Qualitative Results

• Across eight CrossTask tasks, CEFITO’s top-ranked candidate plan matches the ground-truth action sequence in the presented examples.These are illustrative examples selected for qualitative analysis.Source: D Qualitative Results · p. 20
• All top-five candidates are plausible, but incorrect alternatives involve step reordering, action repetition, or substitution with semantically related actions.Examples include reversing “Flip pancake” and “Take pancake from pan,” repeating “Flip steak,” and substituting “Pour water” for “Pour lemon juice.”Source: D Qualitative Results · p. 20
• The modest margin between the correct plan and alternatives suggests CEFITO distinguishes fine-grained semantics and temporal orderings.The examples are illustrative of correctly predicted plans selected for qualitative analysis.Source: D Qualitative Results · p. 20

Failure cases.

• Some failed top-ranked plans still include the correct sequence among the top five candidates, while other failures place it outside the top five.The examples are CEFITO failure cases on CrossTask.Source: Failure cases · p. 20, Figure A.4 · p. 23, Figure A.5 · p. 24
• Many wrong predictions identify the first and last steps but miss intermediate actions, which the model must infer without intermediate visual observations.The model relies only on start and goal observations; examples include “Make lemonade,” “Make pancakes,” and “Make banana ice-cream.”Source: Failure cases · p. 20
• In another failure mode, CEFITO predicts the first action incorrectly but successfully infers the subsequent steps.“Make meringue” is given as an example.Source: Failure cases · p. 20