Source-linked AI summary
ConsensusTAS: Self-Supervised Temporal Action Segmentation for Long-Horizon Construction Videos
Xiaoshan Zhou, Yafei Sun
TL;DR
Long-horizon construction videos require fine-grained action-transition segmentation, but manual temporal-boundary annotation is costly and existing unsupervised methods have important dependencies. ConsensusTAS uses label-free consensus among varied candidate segmentations, achieving strong benchmark results and plausible segmentation of real construction activities.
Problem
Construction activity recognition has largely classified broad categories rather than fine-grained transitions, while temporal boundary annotation for long videos is time-consuming and existing unsupervised methods have several limitations.
Method
ConsensusTAS generates candidates across temporal scales and randomized settings, scores them with internal criteria, and decodes final boundaries from elite-candidate consensus density without labels.
Results
ConsensusTAS achieved the strongest reported unsupervised performance on GTEA and Breakfast, and outperformed several methods on Assembly101 while improving high-overlap localization relative to ASESM.
Takeaways & Limitations
The method produced plausible anonymous phase segmentation in real-world construction video, dividing a 20.9-s bricklaying video into nine phases with density peaks aligned to decoded boundaries.
Abstract
from arXiv · showhide
Recognizing sequential construction activities is important for collaborative human-robot work; for example, robots are able to understand workers' current and upcoming actions and provide timely tool delivery or physical support. However, despite extensive research on construction worker activity recognition, existing studies have been limited to classifying activity categories, such as climbing, lifting, and walking, instead of recognizing fine-grained activity transitions from long-horizon sequences. Addressing this problem is challenging because annotating action temporal boundaries in long construction videos is time-consuming. In this study, we propose ConsensusTAS, a label-free, self-supervised learning approach to segment continuous video streams into distinct activity phases by exploiting the internal consensus of candidate segmentations. We evaluated our algorithm on three public datasets, where it outperformed state-of-the-art methods, achieving an F1@10 of 73.08 on GTEA, an F1@10 of 64.33 on Breakfast, and an F1@50 of 33.50 on static-camera videos from Assembly101. We also tested it on real-world construction videos, where post-hoc evaluation showed that the model successfully recognized and segmented actions within the composite activity of bricklaying, such as spreading mortar on a brick, placing the brick, pressing, and aligning. Compared with other temporal action segmentation models that require computationally intensive large vision-language models, our method can run on a CPU, which provides practical value for video surveillance and human-robot collaboration on mobile robotic platforms.
I. INTRODUCTION
Fine-grained temporal action transitions are needed to support workflow-level construction robotics, but long-video boundary annotation is costly and construction-specific TAS resources are lacking. ConsensusTAS addresses these gaps with a label-free, consensus-based segmentation approach.
- Fine-grained constituent actions and their transitions provide more workflow detail than classifying an entire construction sequence as one activity.The example decomposes hand-sawing into positioning, measuring, grasping, and cutting.
- Frame-level temporal action segmentation is difficult because annotating long videos manually is time-consuming.
- Construction lacks a comparable dataset for natural, multi-step activities and therefore needs label-free methods that partition videos into activity phases.Examples include bricklaying and rebar tying.
- Existing unsupervised TAS methods depend on recurring sub-action patterns, visually salient changes, iterative optimization, initialization, or assumed numbers of latent actions.These dependencies can be problematic with camera motion, occlusion, and environmental changes in construction videos.
- ConsensusTAS generates candidates across temporal scales and randomized settings, scores them with internal representation-based criteria, and decodes boundaries from elite-candidate consensus density.The design reduces dependence on any single initialization or temporal scale while remaining label-free.
A. Problem Formulation
The task is to infer ordered temporal boundaries from frame-level features without action labels. These boundaries create anonymous segments, while preprocessing robustly standardizes and compresses the feature sequence.
- A. Problem Formulation: The input is a video represented as a sequence of frame-level feature vectors.
- A. Problem Formulation: The objective is to infer an ordered set of temporal boundaries within the feature sequence.
- A. Problem Formulation: Prediction and calibration use no action labels.
- A. Problem Formulation: The inferred K boundaries partition the video into K+1 anonymous temporal segments.T denotes sequence length, D feature dimension, and K number of boundaries.
- B. Feature Preprocessing: Each feature dimension is standardized using its temporal median and median absolute deviation.
- B. Feature Preprocessing: Features are centered, projected to at most 32 dimensions with randomized PCA, and L2-normalized per frame.
C. Multiscale boundary evidence
ConsensusTAS measures boundary evidence by comparing left- and right-context representations across sampled temporal scales and smoothing settings. The resulting score combines cosine dissimilarity with Euclidean motion and applies foreground weighting after robust normalization.
- Left- and right-context representations are compared around every frame for each sampled temporal scale and smoothing setting.
- The raw boundary score combines cosine dissimilarity with Euclidean motion.
- After robust normalization, the default foreground weighting is 0.25 + 0.75 MA5( f̂ )t.
D. Candidate Population and Internal Reward
ConsensusTAS evaluates a population of varied candidate segmentations using internal agreement, boundary contrast, compactness, and complexity criteria. Elite candidates are retained for consensus-based decoding.
- The default candidate population contains 48 candidates spanning temporal scales, smoothing widths, projected dimensions, score quantiles, and Gaussian perturbations.Projected dimensions range from 55–100%, score quantiles from 0.72 to 0.92, and perturbation standard deviation is 0.015.
- Candidate agreement is the mean tolerance-based boundary F1 against the remaining candidate population.
- Boundary contrast measures boundary evidence at proposed boundaries, while compactness compares within-segment variance with global variance.
- The top 25% of candidates are retained, with at least three elite candidates.
E. Consensus Density
Consensus density aggregates elite boundary evidence and retains prominent, sufficiently separated peaks to decode temporal segments.
- Elite boundaries contribute Gaussian kernels to the consensus density.
- The density is normalized by its maximum before peak selection.
- Peaks above a dataset/configuration-specific density quantile are retained while enforcing a minimum gap.
III. EXPERIMENTAL PROTOCOL
GTEA provides first-person kitchen videos for temporal action segmentation, with 28 videos across four participants and seven activities.
- GTEA contains 28 first-person kitchen videos recorded with a head-mounted camera.
- Four participants each perform seven activities, including making a sandwich, tea, or coffee.
- The TAS version uses all 28 local feature sequences and four official splits.
2) Breakfast:
Breakfast is a large, varied benchmark for temporal action segmentation, while Assembly101 supplies a static-view subset with coarse assembly labels and 680 activity crops.
- Breakfast: Breakfast records people preparing ten breakfast-related items in 1712 videos from 52 participants.
- Breakfast: Breakfast varies more than GTEA across people, environments, viewpoints, and action sequences.
- Assembly101: The Assembly101 subset C10119 contains 350 usable recordings and 680 assembly/disassembly activity crops.
- Assembly101: Assembly101 coarse labels include attaching or detaching a vehicle component.
- Evaluation: Class-agnostic segment F1 is reported at temporal IoU thresholds τ ∈{0.10, 0.25, 0.50}, excluding background and greedily matching anonymous predicted segments.
IV. RESULTS
ConsensusTAS achieved strong class-agnostic segmentation results on GTEA and Breakfast and performed competitively on Assembly101, with higher-overlap localization benefits and identifiable influential components.
- Benchmark results: On GTEA, ConsensusTAS v1.1 obtained F1 scores of 73.08, 71.07, and 52.40 at overlap thresholds of 10%, 25%, and 50%, respectively.
- Benchmark results: On Breakfast, ConsensusTAS achieved F1 scores of 64.33, 62.45, and 44.73 at overlap thresholds of 10%, 25%, and 50%, respectively.
- Assembly101 results: On Assembly101, ConsensusTAS outperformed uniform segmentation, changepoint detection, KMeans with smoothing, and implemented ASOT variants.
- Assembly101 results: ConsensusTAS had a paired difference of −5.01 at F1@10 but +7.28 at F1@50 relative to the retrospectively best ASESM grid configuration.
- Ablations: Removing density decoding reduced F1@50 from 33.50 to 28.42, while removing internal calibration reduced it to 29.98.
V. REAL-WORLD DEMONSTRATIONS
ConsensusTAS was demonstrated on an unlabeled bricklaying video using a training-free visual front end, producing nine anonymous phases with boundaries supported by consensus density peaks. Medoid frames and post-hoc human review were used to assess the phases’ interpretability and practical plausibility.
- Real-world demonstration: The training-free OpenCV front end sampled frames at 10 frames/s, resized them to 160 × 160, and constructed visual descriptors from HSV, grayscale, edge, and optical-flow features.The descriptor was standardized, projected to 32 dimensions using randomized PCA, and ℓ2-normalized.
- Real-world demonstration: ConsensusTAS divided the 20.9-s bricklaying video into nine anonymous phases.
- Boundary visualization: Consensus boundary-density peaks coincided with decoded phase boundaries, indicating agreement across stochastic runs.
- Phase interpretation: Medoid frames closest to each phase’s mean descriptor summarized the visual content of the inferred phases in temporal order.
- Phase interpretation: A human reviewer evaluated the inferred phases post-hoc for interpretability and practical plausibility.
VI. CONCLUSIONS
The study introduced ConsensusTAS as a label-free approach for temporal action segmentation in long-horizon construction videos. The authors report preliminary benchmark and real-world results, while identifying broader activity coverage and video-benchmark development as future work.
- ConsensusTAS is a label-free approach to temporal action segmentation for long-horizon construction videos.
- Preliminary results showed superior performance on public benchmarks and plausible segmentation of real-world construction activities.
- Future work will scale ConsensusTAS across diverse activities and establish a video benchmark for context-aware human–robot collaboration in long-horizon construction workflows.