Source-linked AI summary
VidParse: Online Parsing of Egocentric Procedures Like a Pro
Anubhav Gupta, Archit Kambhamettu, Vatsal Agarwal, Pulkit Kumar, Abhinav Shrivastava
TL;DR
Online egocentric procedure parsing must convert noisy first-person streams into ordered action steps despite ambiguous local visual evidence and long-range dependencies. VidParse combines manipulation-anchored features, training-free boundary detection, and graph-constrained beam search to enforce procedural structure, achieving competitive performance with strong online baselines while better preserving long-range structure.
Problem
Online parsing must infer current procedural steps from past observations while maintaining coherent state over long sequences.
Method
VidParse uses frozen foundation components for manipulation-anchored features, checkerboard-kernel boundary detection, and beam-search decoding over an induced procedural task graph.
Results
VidParse performs competitively with strong online baselines while improving temporal consistency and preserving long-range procedural structure without training or fine-tuning.
Takeaways & Limitations
N-step transition evaluation measures multi-step action dependencies and supports assessing whether predictions maintain global procedural structure.
Takeaways & Limitations
The decoder may prune practically valid novel action sequences, while severe or prolonged hand-object occlusion can degrade visual evidence and produce spurious predictions.
Abstract
from arXiv · showhide
Translating continuous, noisy egocentric video streams into discrete, temporally ordered action steps is fraught with visual challenges. Heavy ego-motion, transient occlusions, and the high intra-class variability of unscripted human-object interactions cause standard frame-level online temporal models to struggle, often resulting in severe over-segmentation and structural collapse. To bridge the gap between unstable low-level perception and high-level procedural logic, we present VidParse, an online, training-free framework that treats activity understanding as a graph-constrained inference problem. Rather than relying on learned temporal filters, we dynamically identify semantic transitions using a temporal similarity matrix over manipulation-anchored features, which are extracted from frozen foundation models to prioritize foreground hand-object interactions. A beam search decoder then leverages an induced procedural task graph to explicitly enforce valid action transitions and prune impossible trajectories. By anchoring robust visual segments to hard procedural constraints, our approach preserves long-range state transitions and achieves up to a 10x improvement in complex multi-step parsing accuracy over strong online baselines, all without requiring a single gradient update.
1 Introduction
VidParse addresses online egocentric procedure parsing by combining manipulation-focused visual evidence with graph-constrained structured inference. The framework targets coherent, long-range action sequences without task-specific temporal-model training or fine-tuning.
- Online procedure parsing must infer the current action from past observations while maintaining procedural state over long sequences.
- VidParse defines activity understanding as structured inference governed by procedural dependencies between actions.These constraints allow sequences to be inferred when local visual evidence is ambiguous.
- Training-free parsing uses frozen pretrained components, including DINOv2 and hand-object detectors, without task-specific training or downstream fine-tuning.
- Manipulation-Anchored Features combine frozen DINO descriptors with hand-object detections, while a checkerboard kernel detects boundaries from an online temporal similarity matrix.
- Graph-constrained beam search assigns action labels while enforcing valid transitions and maintaining causal inference.
- Across GTEA and EgoPER, VidParse performs competitively with strong online baselines while improving temporal consistency and long-range procedural structure.The evaluation includes an N-step transition metric for multi-step action dependencies beyond standard segmentation metrics.
2 Related Work
VidParse combines frozen interaction-focused representations, training-free online boundary detection, prototype matching, and graph-constrained decoding for causal egocentric procedure parsing. Its task graph imposes procedural structure during inference rather than using learned temporal models or soft graph regularization.
- Online temporal action segmentation: Prior online egocentric segmentation methods rely on causal temporal reasoning from past frames, while existing approaches commonly learn temporal models from annotated data.These dependencies can make deployment difficult when annotated data or retraining is impractical.
- Grammar-based and structural parsing: Graph-constrained beam search enforces valid procedural transitions and removes invalid trajectories by assigning them infinite cost.This hard constraint distinguishes VidParse from prior methods that primarily use graphs as soft training regularizers.
- Prototype-based action matching: Prototype-based matching assigns semantic action labels to segments by comparing their mean descriptors with micro-prototypes.The segment descriptor is matched to the nearest neighbor in the micro-prototype space.
- Problem formulation and structured inference: VidParse formulates online parsing as MAP inference over contiguous temporal segments, assigning labels while respecting a procedural graph.Inference is updated incrementally using a fixed-lag commitment strategy so older predictions cannot be changed by future observations.
- Manipulation-anchored representations: VidParse extracts Manipulation-Anchored Features by masking frozen DINOv2 patch descriptors around detected hands and manipulated objects.The interaction region is projected onto the patch grid, and selected tokens are spatially averaged; missing detections use the last valid feature.
- Similarity-based boundary detection: The method detects action boundaries online from a local temporal similarity matrix using a Gaussian-tapered checkerboard kernel and online peak detection.The detector operates on a short recent-frame buffer, with large novelty values indicating abrupt structural changes that suggest transitions.
4 Experiments
VidParse is evaluated on GTEA and EgoPER, two egocentric procedural benchmarks with different task structures and execution variability. The reported evaluation compares action segmentation against offline and online baselines across procedural datasets.
- Datasets: GTEA contains 28 videos spanning 7 fine-grained kitchen activities and is processed at 15 fps.The benchmark evaluation uses the stated temporal resolution for GTEA.
- Datasets: EgoPER contains 213 normal and 173 erroneous egocentric videos across 5 recipes; evaluation uses only normal videos at 10 fps.The same data splits as the referenced prior work are used.
- Evaluation protocol: The evaluation measures performance under partial observations by parsing progressively larger prefixes containing 10%, 20%, through 100% of each video's frames.The procedure evaluates how parsing behaves as video completion increases.
- Results: Table 1 compares VidParse with offline and online baselines on GTEA and EgoPER and reports state-of-the-art results on all metrics.The table caption describes the comparison as action segmentation on procedural datasets.
- Implementation details: Implementation uses beam width B = 10 for EgoPER and B = 5 for GTEA, with dataset-specific temporal slicing for micro-prototypes.EgoPER uses 4-second segments with a 2-second stride, while GTEA uses 1.5-second segments with a 0.5-second stride.
5 Results
VidParse improves online egocentric procedure parsing through temporally coherent segmentation and structured procedural inference. It preserves longer-range action transitions while remaining training-free and online.
- Action Segmentation Performance: VidParse improves F1 by more than 15% across thresholds, with up to 25% gains on GTEA F1@0.5.On EgoPER, it improves F1 and Edit metrics by roughly 25% while gaining 4% in frame-level accuracy.
- Procedural Parsing Accuracy: VidParse consistently outperforms ProTAS across transition lengths, with 5-step and 7-step AUC up to 5×–10× higher.The widening gap indicates stronger preservation of global task structure than frame-level baselines.
- Boundary Analysis: Approximately 75% of predicted segments last 2–4 seconds, and more than 90% are shorter than 5 seconds.These segment durations characterize the temporal behavior of the online boundary detector.
- Action Segmentation Performance: Segment-level processing produces temporally stable predictions that avoid the rapid label fluctuations commonly observed near action boundaries.Figure 3 shows smoother transitions that closely follow the ground-truth structure.
- Ablation Studies: Table 3 compares beam width, inference speed, and feature representations, with the CPU-based method reported as faster than GPU-accelerated ProTAS.Frame-level MAFs are emphasized over clip-level and full-frame features.
- Ablation Studies: Manipulation-Anchored Features outperform alternative representations, while the centroid-based prototype method reaches 80.69 Accuracy and 77.61 F1@0.5.Varying prototypes per action class from 3 to 11 changes Accuracy and Edit by less than 1.2%.
6 Limitations
VidParse has latency, generalization, detector-robustness, and modality limitations. Its segment-level processing may not suit ultra-low-latency applications, and performance can degrade under weak manipulation cues.
- Limitations: Segment-level processing may be unsuitable for ultra-low-latency applications, while the learned task graph may generalize poorly to unseen procedural variations.The hand-object detector can also fail or lag under severe occlusion, motion blur, or non-interactive actions.
- Limitations: Performance may degrade on exocentric videos when manipulation cues are weak or absent.The framework may extend to exocentric settings when those cues remain visible.
7 Conclusion
VidParse is presented as an online, training-free framework that combines foundation-model features with procedural constraints to preserve long-range structure in egocentric procedure parsing.
- VidParse treats activity understanding as structured inference for online egocentric procedure parsing.
- Manipulation-Anchored Features and a Gaussian-tapered checkerboard kernel over frozen DINOv2 representations detect action boundaries without gradient-based learning.
- Task-graph-constrained beam search enforces procedural consistency during parsing.
- Gap rectification corrects transient visual failures while structural priors and foundation-model features preserve long-range procedural structure.
- The experiments show competitive performance with strong online baselines.
Supplementary Material for: VidParse: Online Parsing of Egocentric
The supplementary material identifies the authors and their affiliation with the University of Maryland, College Park, USA.
- The listed authors are Anubhav Gupta, Archit Kambhamettu, Vatsal Agarwal, and Pulkit Kumar.
- Abhinav is also included in the listed University of Maryland contact addresses.
- The authors are affiliated with the University of Maryland, College Park, USA.
1 Qualitative Structured Parsing Results
The qualitative supplementary results provide interactive parses for several EgoPer recipes, pairing real-time segmentation with synchronized task-graph visualizations.
- Interactive qualitative results include complete video parses for Coffee, Tea, Oatmeal, Quesadilla, and Pinwheels in EgoPer.
- Viewers can observe real-time action segmentation for each sequence.
- The visualizations synchronize the action segmentation with the underlying task graph.
- The task-graph visualization demonstrates pruning of invalid transitions by VidParse.
2 Task Graphs
Recipe-specific task graphs encode atomic actions, observed transitions, prerequisite structure, optionality, terminal steps, and repeated behavior for procedural parsing.
- Task Graphs: Each recipe uses a directed graph G = (V, E), whose nodes represent atomic actions and whose edges represent observed sequential transitions.
- Task Graphs: Fig. 2 visualizes task-graph structural nuances using node colors and edge styles.
- Node Definitions (Actions/Steps): Green start nodes represent actions that require no prior prerequisite steps.
- Node Definitions (Actions/Steps): Red end nodes are terminal actions that conclude at least one video sequence for a recipe.
- Node Definitions (Actions/Steps): Yellow optional nodes are omittable actions appearing in some but not all recipe videos, while gray or white nodes are mandatory intermediate actions present in all observed sequences.
- Edge Definitions (Transitions): Solid black edges denote first-visit paths, whereas dashed red edges denote revisit paths for corrections, loops, or alternative chronological pathways.
3 Feature Representation Ablation on GTEA
On GTEA, recipe-specific task graphs expose a large gap between feature quality alone and strict graph-constrained inference. MAFs consistently perform best, while VidParse substantially exceeds the adapted ProTAS baseline.
- MAFs consistently outperform other feature representations across methods in the recipe-specific GTEA task-graph setting.The evaluation isolates recipes such as Cheese, Coffee, and Hotdog to test fine-grained, object-aware procedural reasoning.
- The comparison separates the contribution of strong hand-object features from the benefit of structurally constrained decoding for exact procedural trajectories.
- 21.30 overall F1 is achieved by ProTAS adapted with recipe-specific structure and MAF representations.
- 80.47 overall F1 is achieved by VidParse using the same MAFs with strict graph-constrained inference.VidParse also achieves near-perfect structural parsing on recipes such as Cheese.
4 Visualizing Boundary Detection
VidParse detects online boundaries from temporal similarity patterns over manipulation-anchored features, while visibility-aware decoding handles steps with missing hand-object evidence. The design remains bounded by graph and detector failures.
- Boundary Detection: Temporal Similarity Matrix visualization shows continuous procedural actions as distinct, highly correlated block-diagonal structures over tea-making video.These structures arise from the semantic stability of frozen DINOv2 features used by MAFs.
- Visibility Prior: A visibility prior penalizes segments without reliable hand-object interaction evidence, filtering background noise during parsing.
- Background Suppression: Recipe-specific non-manual actions can temporarily zero the hand-visibility penalty, preserving continuity when hands leave the frame.The suppression state is lifted when hand-object interactions become highly visible again; reported use is limited to Microwave for X seconds in EgoPer.
- Limitations: The induced task graph can prune practically valid but novel action sequences, while prolonged occlusion or purely non-manual actions degrade visual evidence.In extended failure cases, the model may emit spurious predictions or rely heavily on the last valid feature state.