PaperlayerView exact revision on arXiv

Source-linked AI summary

Segmental Spatiotemporal CNNs for Fine-grained Action Segmentation

Colin Lea, Austin Reiter, Rene Vidal, Gregory D. Hager

arXiv:1602.02995v4cs.CVcs.RO

TL;DR

Fine-grained action segmentation remains difficult because subtle object states, relationships, and temporal changes are not captured well by conventional contextual cues. The paper combines a spatial-temporal CNN with a semi-Markov segmental model and faster constrained inference. It substantially outperforms recent methods on cooking and surgical datasets, while segmental smoothing can hurt short-duration actions.

  • Problem

    Fine-grained recognition remains limited because background and holistic cues do not capture subtle object states, locations, and relationships needed for action segmentation.

  • Method

    The method combines an ST-CNN for object and temporal relationship features with a semi-Markov model for segment transitions and constrained segmental inference.

  • Results

    The full model has 27.8% better accuracy on 50 Salads and 37.6% better accuracy on JIGSAWS relative to the IDT baseline.

  • Takeaways & Limitations

    The approach substantially outperforms Dense Trajectories, pre-trained spatial CNNs, and LSTM-based temporal models while computing features, training, and inference more efficiently than relevant alternatives.

  • Takeaways & Limitations

    The segmental model can smooth over short actions, and the authors identify duration cues as a direction for future work.

Abstract

from arXiv · show

Joint segmentation and classification of fine-grained actions is important for applications of human-robot interaction, video surveillance, and human skill evaluation. However, despite substantial recent progress in large-scale action classification, the performance of state-of-the-art fine-grained action recognition approaches remains low. We propose a model for action segmentation which combines low-level spatiotemporal features with a high-level segmental classifier. Our spatiotemporal CNN is comprised of a spatial component that uses convolutional filters to capture information about objects and their relationships, and a temporal component that uses large 1D convolutional filters to capture information about how object relationships change across time. These features are used in tandem with a semi-Markov model that models transitions from one action to another. We introduce an efficient constrained segmental inference algorithm for this model that is orders of magnitude faster than the current approach. We highlight the effectiveness of our Segmental Spatiotemporal CNN on cooking and surgical action datasets for which we observe substantially improved performance relative to recent baseline methods.

1 Introduction

Fine-grained action recognition requires modeling object states, relationships, and temporal changes rather than relying on background or holistic cues. The proposed system combines a spatiotemporal CNN, a segmental semi-Markov model, and faster constrained inference, improving results on cooking and surgical datasets.

  • Motivation: Fine-grained actions depend on subtle object-location and state changes that background appearance cannot adequately capture.This limitation helps explain why progress has lagged behind large-scale action recognition.
  • Motivation: The task is to temporally segment a video and classify each constituent action in situated, goal-driven activities such as salad preparation.The targeted setting includes long sequences recorded by a static camera, with applications in manufacturing, surgical training, and activity analysis.
  • Approach: The ST-CNN captures object locations, states, and inter-object relationships, while a semi-Markov model represents high-level temporal information and action transitions.Its temporal filters model how relationships change during actions, such as a tomato changing from whole to diced or moving between locations.
  • Approach: A constrained segmental inference algorithm is one to three orders of magnitude faster than the common semi-Markov inference technique.The speedup addresses the higher computational complexity typically associated with semi-Markov inference.
  • Evaluation: The model is evaluated on the 50 Salads cooking dataset and the JIGSAWS surgical robotics dataset, substantially outperforming recent fine-grained recognition methods.The datasets provide task granularity, realistic variability, and sufficient data for joint segmentation and classification.

2 Related Work

Prior work spans holistic trajectory representations, CNN-based spatial and temporal features, pose and object-centric methods, and sequential models such as CRFs and LSTMs. The paper positions its model as combining interpretable object-relationship features with segment-level transition modeling.

  • Holistic and CNN Features: Holistic IDT representations are standard baselines, but large-scale CNN approaches often remain only marginally better than spatial-only CNNs or IDT.Some CNN systems exceed holistic baselines only after concatenating CNN and IDT features.
  • From Classification to Segmentation: Large-scale recognition methods perform much better on classification than localization, illustrating the difficulty of transferring classification success to fine-grained temporal segmentation.The THUMOS comparison reports 70% mAP for classification versus 18% for localization at overlap ≥0.5.
  • Fine-grained Recognition: Fine-grained methods add pose, object, hand, or egocentric cues, but object-centric approaches can require many manual object annotations.The proposed work instead learns a latent object representation without object annotations.
  • Temporal and Segmental Models: CRFs combine segment-wise unary terms with pairwise action transitions, whereas this work applies such segmental modeling to learned spatiotemporal CNN features.Earlier CRF applications often modeled segments with histograms of holistic features.
  • Temporal and Segmental Models: LSTM-based RNNs implicitly learn latent transitions but are difficult to interpret, while the proposed temporal CNN explicitly models and visualizes temporal state changes.The comparison highlights interpretability as a distinction between the approaches.

3 Spatiotemporal CNN Model

The ST-CNN combines spatial processing of video frames with long-range temporal convolutions to represent fine-grained actions. It is trained with frame-level auxiliary labels and uses learned activations and classifiers to produce action probabilities over time.

  • Model input and output: The model processes a color image and motion image for each frame, producing an action-probability vector at every frame.Motion is represented with a Motion History Image computed from frame differences over a 2-second window.
  • Spatial component: The spatial CNN uses hierarchical 3 × 3 convolutional filters, ReLUs, and max pooling to capture object texture and spatial location.Each spatial unit divides activations into regions, with region features concatenated before a fully connected layer.
  • Spatial component: The fully connected spatial representation captures relationships between regions and latent object representations, such as tomato, cutting board, and knife configurations.The paper describes these representations as state vectors derived from activations across all spatial regions.
  • Training: The spatial component is trained using frame-level auxiliary action labels, while its state vector h_t is passed to the temporal component.The auxiliary softmax predictions are used only to train the spatial component; the temporal component receives h_t as input.
  • Temporal component: The authors use long temporal filters because they capture mid-range motion patterns, although adding multiple temporal layers did not improve preliminary performance.Network parameters are learned with cross-entropy losses, independent spatial and temporal optimization, ADAM, dropout, and cross-validation.

4 Segmental Model

The segmental model jointly segments and classifies actions by combining ST-CNN activations with pairwise transitions in a semi-Markov formulation. Constrained inference limits segment counts, avoids gross over-segmentation, and is substantially faster than traditional semi-Markov inference.

  • Segmental Model: The model uses ST-CNN activations and pairwise action-transition terms to jointly segment and classify actions.The semi-Markov formulation represents action segments explicitly and conditions each segment on the preceding segment.
  • Segmental Model: Each action segment is represented by its label, start time, and duration, with consecutive segments covering the video.The segment durations sum to the total video time.
  • Segmental Model: The segment score combines ST-CNN frame scores with a transition score between consecutive action labels.Transition probabilities are estimated from training data as logarithms of action-to-action probabilities.
  • Segmental Inference: Constrained inference bounds the number of segments by K and removes redundant duration variables from the optimization.The resulting formulation uses only increasing segment start times, with durations recovered from adjacent starts.
  • Segmental Inference: The dynamic program recursively computes the best segmentation for 1 through K segments and recovers the optimal labeling by backtracking.Its forward pass evaluates segment counts, times, and action classes.
  • Segmental Inference: The proposed algorithm has complexity O(KTC^2), is K times more efficient than Segmental Viterbi when K < D, and achieved one-to-three-order-of-magnitude speedups.It uses O(KTC) memory versus O(TC) for Segmental Viterbi, with K set from the training split’s maximum segment count.

5 Experimental Setup

The experiments evaluate fine-grained action segmentation and classification on cooking and surgical datasets using segmental and frame-wise metrics. The setup includes spatial and temporal baselines, multiple 50 Salads granularities, and an inference-speed analysis.

  • Datasets: 50 Salads contains 50 salad-preparation instances annotated at four granularities, including 17 mid-level and 51 low-level actions.The coarsest labels cover broad activities, while finer labels distinguish specific actions and their phases.
  • Datasets: The study evaluates 50 Salads for cooking and JIGSAWS for surgical robotics, both selected for task granularity, data volume, and realistic variability.JIGSAWS evaluation uses the suturing task with synchronized video and robot-kinematics data.
  • Metrics: Evaluation uses segmental metrics for action ordering and cohesion, frame-wise accuracy for sequence coverage, and segment classification accuracy when temporal boundaries are known.The segmental score is based on normalized Levenshtein edits and is reported on a 0–100 scale.
  • Baselines: Baselines include IDT and pretrained VGG spatial representations, plus an LSTM temporal model using the spatial CNN as input.Known start and end times are supplied for segment classification results.
  • Reported Analyses: The experiments report results in Tables 1–3 for 50 Salads and JIGSAWS, including a dedicated 50 Salads granularity analysis.Table 1 covers the eval granularity, Table 2 covers JIGSAWS, and Table 3 compares four 50 Salads granularities.

6 Results & Discussion

The experiments show that spatiotemporal and segmental modeling improve fine-grained action recognition, with gains varying by dataset and metric. The full model substantially improves accuracy over IDT, while temporal modeling improves edit scores and segmental modeling can smooth short actions away.

  • 27.8% better accuracy on 50 Salads and 37.6% better accuracy on JIGSAWS relative to the IDT baseline.
  • Spatial Model: The spatial model is at least 12% better than IDT and 21% better than VGG on both datasets, although edit scores remain very low.Local temporal information produces oscillations in predictions.
  • Temporal Model: ST-CNN outperforms S-CNN on both datasets, with large temporal filters substantially improving edit score while also smoothing predictions.The filters capture different phases of an action, such as its start or finish.
  • Segmental Model: The segmental model notably improves JIGSAWS but only modestly improves 50 Salads.It helps oscillating predictions but can smooth over actions that are short in duration.
  • Action Granularity: Performance decreases as the number of action classes increases, but results degrade sublinearly with each additional class.
  • Other Results: Instrumented kitchen tools achieve 58.46% edit score and 81.75% accuracy on 50 Salads, while video is described as more practical but harder to model.
  • 90.47% classification accuracy on JIGSAWS exceeds 81.17% for the video-based linear dynamical system and 86.56% for its video-and-kinematics hybrid.The improvement for joint segmentation and classification is modest, and position and velocity information may be especially effective for these surgical actions.
  • The inference algorithm scales readily to full-length videos through speedups over Segmental Viterbi.

7 Conclusion

The paper concludes that its segmental spatiotemporal CNN substantially outperforms several popular action-recognition baselines. It also reduces feature-computation and training time relative to IDT and LSTM and performs inference more efficiently than traditional segmental methods.

  • The segmental spatiotemporal CNN substantially outperforms Dense Trajectories, pre-trained spatial CNNs, and LSTM-based temporal models.
  • The approach computes features faster than IDT, trains faster than LSTM, and performs inference more efficiently than traditional segmental methods.
Loading 1602.02995v4…