VidParse: Online Parsing of Egocentric Procedures Like a Pro

• Online parsing must infer each current step from past observations alone, making coherent procedural-state estimation over long sequences a central challenge.Source: §1 · p. 1
• Existing learned temporal approaches rely on annotated data and learned dynamics, making them sensitive to domain shifts and difficult to deploy when data or retraining is impractical.Source: §1 · pp. 1–2
• VidParse combines Manipulation-Anchored Features from frozen DINO descriptors and hand-object detections, checkerboard-kernel boundary detection over an online temporal similarity matrix, and graph-constrained beam search.The beam search selects action sequences using visual evidence and procedural constraints while maintaining causal inference.Source: §1 · pp. 2–3
• VidParse is evaluated on GTEA and EgoPER, where it performs competitively with strong online baselines despite using no training or fine-tuning and improves temporal consistency.Source: §1 · p. 3
• VidParse introduces an N-step transition evaluation of long-range procedural consistency that measures multi-step action dependencies beyond standard segmentation metrics.The paper reports that VidParse preserves coherent multi-step trajectories more reliably than prior online approaches.Source: §1 · p. 3

2.1 Temporal Action Segmentation

• Causal egocentric segmentation faces over-segmentation; prior fixes use adaptive memory, Mamba-OTR, and test-time vision-language adaptation [15], whereas VidParse uses frozen features without temporal optimization.The section describes a shift from offline temporal action segmentation to causal online segmentation for real-time egocentric applications.Source: §2.1 · pp. 3–4

2.2 Similarity-Based Boundary Detection

• Prior GEBD methods include ESTimator, DyBDet, DiffGEBD, UBoCo, OTAS, and ATBA; VidParse instead applies a checkerboard kernel to frozen DINOv2 [31] object-centric features.Unsupervised and weakly supervised alternatives still require substantial feature or decoder optimization; the paper presents its approach as the first training-free GEBD integration into graph-constrained action parsing.Source: §2.2 · pp. 4–5

2.3 Grammar-Based and Structural Parsing

• Prior procedural parsers explore learned graphs, grammars, and language-model logic [17] [16], while VidParse uses an induced task graph as a hard beam-search constraint against invalid transitions.Earlier graph-based approaches often use graphs as soft training regularizers, and many procedure-learning and alignment methods are offline; VidParse assigns infinite cost to invalid transitions to prevent structural collapse.Source: §2.3 · p. 5

3.1 Problem Formulation

• Online parsing is formulated as assigning labels to contiguous video segments while respecting the task’s procedural structure.The objective is MAP inference over the ordered action sequence given the segmented stream and procedural graph.Source: §3.1 · p. 6

3.2 Manipulation-Anchored Features (MAFs)

• MAFs focus frozen foundation-model features on hand–object interactions, suppressing background clutter and smoothing frames without detections.DINOv2 patch tokens are pooled within a region localized by a frozen hand-object detector; when detections are absent, the last valid feature is propagated.Source: §3.2 · pp. 6–7

3.3 Training-Free Boundary Detection

• A training-free online novelty detector finds action boundaries from local temporal similarity patterns in the MAF stream.It applies a Gaussian-tapered checkerboard kernel to a sliding-window similarity matrix, then emits online novelty peaks subject to a threshold and minimum boundary spacing.Source: §3.3 · pp. 7–8

3.4 Prototype-Based Action Matching

• Action labels are assigned by matching each interaction-filtered segment to the nearest action micro-prototype.Training examples are clustered by execution style and sliced into overlapping short windows, preserving local temporal phases rather than averaging an entire action into one prototype.Source: §3.4 · p. 8

3.5 Structured Procedural Inference

• Graph-constrained decoding combines visual, manipulation, and procedural evidence to select valid action sequences while handling brief visibility gaps.The task graph is induced from training sequences using observed transitions, prerequisites, omittable steps, and start nodes; fixed-lag commitment keeps inference online. During low visibility, the decoder extends an active action or assigns background, and can rectify a gap when the preceding action returns with high confidence.Source: §3.5 · pp. 9–10

4.1 Datasets and Metrics

• Evaluation uses GTEA’s 28 videos across 7 kitchen activities and EgoPER’s normal videos across 5 recipes, with Accuracy, Edit distance, and F1 at overlap thresholds 0.1, 0.25, and 0.5.GTEA is processed at 15 fps; EgoPER at 10 fps, using the same splits as.Source: §4.1 · p. 10
• N-Step Transition Accuracy measures multi-step dependency consistency using transition precision–recall AUC across video completion levels from 10% to 100%.The metric compares n-step transitions in predicted and ground-truth sequences to assess procedural-state coherence as evidence accumulates.Source: §4.1 · p. 10

4.2 Implementation Details

• Implementation averages DINOv2 patch tokens intersecting hand-object interaction regions, then applies training-free boundary detection and prototype matching with dataset-specific settings.The frozen backbone is ViT-L/14 [31], and the hand-object detector is from. Checkerboard widths are 20 frames for EgoPER and 10 for GTEA; cluster counts are 3 for GTEA and 9 for EgoPER. Micro-prototype segments use 4-second lengths and 2-second strides for EgoPER, versus 1.5-second lengths and 0.5-second strides for GTEA; beam widths are 10 and 5, respectively.Source: §4.2 · pp. 10–11

4.1 Datasets and Metrics

• The supplied Table 1 caption identifies an action-segmentation comparison with offline and online baselines on GTEA and EgoPER, but provides no scores to report.The caption alone does not establish a winner or numerical result.Source: Table 1 · p. 11

5.1 Action Segmentation Performance

• F1 improves by more than 15% across thresholds and up to 25% at F1@0.5 on GTEA; EgoPER gains roughly 25% on F1 and Edit and 4% in frame accuracy.Predictions remain smooth and closely follow ground-truth segmentation, without the flickering seen in frame-level models.Source: §5.1 · p. 11, Figure 3 · p. 12

5.2 Procedural Parsing Accuracy

• AUC for 5-step and 7-step transitions is up to 5×–10× higher than ProTAS, and VidParse outperforms it across transition lengths while preserving global task structure.Frame-level baselines accumulate local errors that disrupt long-range transitions and violate procedural constraints; VidParse reconstructs the task graph rather than recognizing actions in isolation.Source: Figure 4 · p. 12, §5.2 · pp. 12–13

5.3 Online Boundary Detection Behavior

• 75% of segments fall within a 2–4 second window and more than 90% are shorter than 5 seconds; at kernel=10, boundaries are recovered within 2 seconds over 90% of the time.The method achieves the best downstream action-segmentation performance among the compared online unsupervised boundary methods, and segment-level predictions remain temporally stable.Source: Figure 5 · p. 13, §5.3 · p. 13

5.4 Ablation Studies

• MAF features improve segmentation metrics over ProTAS; full-frame features underperform MAFs, frame-level EgoVLPv2 remains competitive with DINOv2 MAFs, and clip-level features perform worse.The comparisons indicate that foreground information helps and coarse temporal representations can miss short procedural transitions.Source: §5.4 · pp. 13–14, Table 2 · p. 14, Table 3 · p. 14
• VidParse runs at 8.6 FPS versus ProTAS at 4.5 FPS, and CPU parsing and decoding take 32.8s versus 706s for ProTAS on an NVIDIA RTX A4000.The minimum boundary-gap parameter d trades denser boundaries that may mildly over-segment actions against wider spacing that may miss rapid transitions.Source: Table 3 · p. 14, §5.4 · p. 14
• Centroid-based prototypes achieve the best results across metrics, with Acc. 80.69 and F1@0.5 77.61; varying prototypes per class from 3 to 11 changes Accuracy and Edit by less than 1.2%.Global means collapse the action feature space, while medoids alone limit representations to previously observed instances.Source: §5.4 · p. 14

6 Limitations

• Segment-level processing may be unsuitable for ultra-low-latency applications, and the learned task graph may generalize poorly to unseen procedural variations.The framework does not require global context, but these limitations remain.Source: §6 · p. 15
• Under severe occlusion, motion blur, or non-interactive actions, the Hand-Object Detector can produce incorrect or delayed predictions.The passage gives “Using a Microwave” as an example of a non-interactive action.Source: §6 · p. 15
• The framework may extend to exocentric videos when manipulation cues are visible, but performance may degrade when those cues are weak or absent.Source: §6 · p. 15
• Future work could use probabilistic graphs to better handle open-world and loosely structured tasks.Source: §6 · p. 15

7 Conclusion

• VidParse achieves competitive performance with strong online baselines while better preserving long-range procedural structure.Experiments attribute this result to combining structural priors with foundation model features.Source: §7 · p. 15
• Manipulation-Anchored Features and a Gaussian-tapered checkerboard kernel over frozen DINOv2 representations detect action boundaries without gradient-based learning.Source: §7 · p. 15
• Task-graph-constrained beam search with gap rectification enforces procedural consistency and corrects transient visual failures.Source: §7 · p. 15

1 Qualitative Structured Parsing Results

• The supplementary page shows complete parses for Coffee, Tea, Oatmeal, Quesadilla, and Pinwheels, where the task graph visualizes how VidParse prunes invalid transitions.Each sequence pairs real-time action segmentation with a synchronized task-graph visualization.Source: §1 · p. 19

2 Task Graphs

• Each recipe graph aggregates action sequences across its videos, with nodes for atomic steps and directed edges for observed sequential transitions.The graphs account for skipped steps, repeated actions, and differing end states.Source: §2 · p. 19
• Node colors distinguish prerequisite-free start actions, actions concluding at least one sequence, omittable actions, and mandatory intermediate actions.Optional actions appear in some but not all sequences; standard intermediate actions appear in all observed sequences.Source: §2 · p. 20
• Solid black edges mark first-visit transitions to new actions, while dashed red edges mark revisits such as corrections, loops, or alternative chronological paths.The figures visualize these structural nuances with node colors and edge styles.Source: §2 · p. 20

3 Feature Representation Ablation on GTEA

• Table 1 evaluates VidParse and ProTAS across isolated GTEA recipes under recipe-specific task graphs to assess fine-grained, object-aware reasoning.The evaluated recipes include Cheese, Coffee, and Hotdog.Source: §3 · p. 20, Table 1 · p. 23
• MAFs consistently outperform other representations because their hand-object interaction features suit task graphs that require precise object context.Recipe-specific task graphs couple each action to the manipulated object, enforcing precise sequential constraints.Source: §3 · p. 20, Table 1 · p. 23
• 80.47 overall F1 for VidParse versus ProTAS’s 21.30 when both use MAFs and recipe-specific task graphs on GTEA.ProTAS plateaus at 21.30 overall F1, while VidParse uses the same MAFs with strict graph-constrained inference to reach 80.47.Source: §3 · pp. 22–23, Table 1 · p. 23

4 Visualizing Boundary Detection

• High cosine similarity forms block-diagonal structures for frames within the same action, while differing steps show low similarity in the Temporal Similarity Matrix (TSM).The features rely on the semantic stability of a frozen DINOv2 backbone.Source: §4 · p. 23, Figure 3 · p. 24
• Hand-object state changes produce sharp drops in local temporal similarity, which VidParse uses to partition continuous video into semantically coherent segments without learned temporal filters.The passage gives transitions such as moving from placing a tea bag to pouring hot water as examples.Source: §4 · p. 23

5 Action conditioned background suppression

• Because tasks such as waiting or baking can involve extended periods without visible hands, VidParse temporarily suppresses the visibility penalty for predefined recipe-specific non-manual actions.The visibility prior otherwise penalizes segments lacking reliable hand-object interaction evidence.Source: §5 · p. 23
• During suppression, the decoder avoids spuriously transitioning to BG by zeroing the hand-visibility penalty and extrapolating beam transition cost from the action’s accumulated cost rate.Suppression lifts when hand-object interactions become highly visible again.Source: §5 · pp. 23–24
• In the reported EgoPer results, action-conditioned suppression was applied only to “Microwave for X seconds.”The mechanism is described as action-conditioned inertia, with standard visual evidence driving inference again after suppression ends.Source: §5 · p. 24

6 Failure Cases and Limitations

• Infinite-cost task-graph violations can prune a correct sequence when a user performs a novel but valid action sequence, forcing alignment to the known graph.The structured prior is induced directly from training sequences.Source: §6 · pp. 24–25
• Dependence on the Hand-Object Detector leaves prolonged severe occlusion or purely non-manual actions vulnerable to degraded evidence, spurious predictions, or reliance on the last valid feature state.Gap rectification and inertia address transient occlusions, but not these extended failure cases.Source: §6 · p. 25