Source-linked AI summary
Actor and Action Video Segmentation from a Sentence
Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, Cees G. M. Snoek
TL;DR
The paper addresses pixel-level actor-action segmentation from language rather than fixed actor-action vocabularies. It introduces a video-optimized encoder-decoder model with sentence-conditioned segmentation and reports feasibility, robustness, generalization, and improved traditional segmentation performance.
Problem
Existing actor-action segmentation methods use fixed actor-action pair vocabularies, limiting sentence-guided segmentation of fine-grained actors, instances, and out-of-vocabulary pairs.
Method
The paper proposes a fully convolutional, end-to-end trainable encoder-decoder that combines language and video representations for sentence-guided pixel-level segmentation.
Results
Experiments demonstrate sentence-guided segmentation quality, generalization ability, and an advantage over the state-of-the-art on traditional actor-action segmentation.
Takeaways & Limitations
Extending A2D and J-HMDB with more than 7,500 sentence descriptions supports studying actor-action segmentation from natural-language queries.
Abstract
from arXiv · showhide
This paper strives for pixel-level segmentation of actors and their actions in video content. Different from existing works, which all learn to segment from a fixed vocabulary of actor and action pairs, we infer the segmentation from a natural language input sentence. This allows to distinguish between fine-grained actors in the same super-category, identify actor and action instances, and segment pairs that are outside of the actor and action vocabulary. We propose a fully-convolutional model for pixel-level actor and action segmentation using an encoder-decoder architecture optimized for video. To show the potential of actor and action video segmentation from a sentence, we extend two popular actor and action datasets with more than 7,500 natural language descriptions. Experiments demonstrate the quality of the sentence-guided segmentations, the generalization ability of our model, and its advantage for traditional actor and action segmentation compared to the state-of-the-art.
1. Introduction
The paper introduces sentence-guided pixel-level segmentation of an actor and its action in video, replacing fixed actor-action pair vocabularies with natural-language queries. It proposes a video-optimized fully convolutional architecture and expands two datasets with sentence annotations.
- The paper defines actor and action segmentation from a sentence as pixel-level video segmentation guided by a natural-language input.
- Unlike prior methods using fixed actor-action pairs, the approach accepts natural-language descriptions to specify the actor and action.
- The proposed model is a fully convolutional, end-to-end trainable encoder-decoder architecture optimized for video.
- The authors extend A2D and J-HMDB with more than 7,500 textual sentences describing actors and actions in video content.
- Experiments assess sentence-guided segmentation quality, model generalization, and performance on traditional actor-action segmentation.
2. Related Work
Prior actor-action segmentation methods operate over fixed label-pair vocabularies, whereas this work uses language as an open-set representation for pixel-level actor and action segmentation. It extends sentence-based video localization toward diverse actors, actions, and instances.
- The A2D dataset and related methods assign video regions labels from a fixed vocabulary of 43 actor-action pairs.
- The proposed model represents the joint actor-action space with an open set of labels as rich as language.
- Language-guided segmentation can distinguish fine-grained actors within a super-category and identify different actor and action instances.
- Pre-trained word embeddings allow inference from words outside the actor-action vocabulary when those words exist in the embedding.
- Compared with sentence-based person retrieval and localization, this work targets any actor and outputs pixel-wise segmentation of both actor and action.
- Unlike sentence-based temporal action retrieval or zero-shot tube localization, the paper studies spatio-temporal segmentation from language.
3. Model
The model encodes language and video separately, generates sentence-conditioned dynamic filters, and decodes their interactions into pixel-level segmentation maps. Video-aware features, multi-resolution prediction, and deconvolution support the final mask.
- The architecture combines a textual encoder, a 3D video encoder, and a decoder for pixel-wise segmentation.
- Textual encoder: The textual encoder uses pre-trained word2vec embeddings and a 1D convolution with ReLU and max-pooling to represent the sentence.
- Video encoder: The video encoder uses 3D convolutional features to preserve spatial information while modeling temporal video dynamics.
- Dynamic filtering: Dynamic filters generated from the sentence representation are convolved with video representations at multiple resolutions to produce segmentation response maps.
- Decoder: A deconvolutional network operates directly on the video representation to produce full-resolution masks, better handling small objects and smoother predictions.
- Training: Training uses binary masks at resolutions 32, 128, and 512, combining their weighted losses to improve gradient flow.
4. Datasets
The paper augments A2D and J-HMDB with natural-language descriptions of the actors and actions in their videos. The resulting annotations add discriminative, fine-grained language for sentence-guided segmentation.
- A2D Sentences: A2D contains 3,782 videos with pixel-level actor and action labels spanning eight actions and seven actor classes.The dataset is split into 3,036 training videos and 746 testing videos.
- A2D Sentences: 6,656 A2D sentences use discriminative expressions to identify actor instances and include 811 nouns, 225 verbs, and 189 adjectives.These annotations refine actor-action pairs with more granular descriptions, such as man, woman, or player for adult.
- A2D Sentences: A2D sentences average 7.3 words, or 4.5 content words without function words, compared with 4.7 and 3.6 in ReferIt.The richer descriptions reflect the inclusion of varied verbs, which referring-expression datasets often omit.
- J-HMDB Sentences: J-HMDB provides 928 clips covering 21 actions, and the paper adds one natural-language target description to each video.The J-HMDB annotations contain 158 nouns, 53 verbs, and 23 adjectives and follow the A2D annotation protocol.
- Dataset release: The authors provide sentence-annotated examples for both datasets and state that the annotations and model code will be released.Examples appear in Figure 3, with additional dataset details in the supplemental material.
5. Experiments
Experiments evaluate the model through ablations, sentence-guided segmentation against baselines, generalization to J-HMDB Sentences, and traditional actor-action segmentation on A2D.
- Ablation Study: Using 16 input frames raises overall IoU from 48.2% with one frame to 53.6%, highlighting the value of temporal context.The evaluated settings are N = 1, 4, 8, 16, with overall IoU values of 48.2%, 52.2%, 52.8%, and 53.6%, respectively.
- Ablation Study: The convolutional textual encoder reaches 53.6% overall IoU, compared with 51.8% for LSTM and 52.1% for bidirectional LSTM.The model therefore uses the convolutional encoder in subsequent experiments.
- Ablation Study: Multi-resolution loss achieves 53.6% overall IoU versus 49.4% when only the highest resolution is used.The compared resolutions are 32, 128, and 512, with all three contributing in the multi-resolution setting.
- Ablation Study: RGB-flow fusion with RGB weighted twice as strongly as Flow reaches 55.1% overall IoU, exceeding RGB-only performance at 53.6%.RGB alone scores 53.6%, while Flow alone scores 49.5%, using 16 frames for both streams.
- Segmentation from a sentence: On A2D Sentences, the model outperforms both baselines across all metrics, with RGB inputs improving mAP by 3.5%, overall IoU by 2.1%, and mean IoU by 6.7%.Fusion of RGB and Flow further improves results, and the larger mean-IoU gain indicates stronger performance on small objects.
- Segmentation from a sentence: The model generalizes from A2D Sentences to J-HMDB Sentences without additional fine-tuning and produces more accurate segmentations at higher overlap values.J-HMDB Sentences contains human actions, with four of its 21 actions overlapping those in A2D Sentences.
- Segmentation from actor and action pairs: On A2D, the method improves state of the art by 4.9% in class-average accuracy and 5.1% in Mean IoU for joint actor-action segmentation.For action segmentation, it improves class-average accuracy by 8.8% and Mean IoU by 7.2%.
6. Conclusion
The paper introduces sentence-guided actor and action video segmentation and an encoder-decoder architecture that accounts for video’s spatio-temporal structure. Experiments support feasibility, robustness, adaptation to semantic segmentation of actor-action pairs, and improved state-of-the-art performance.
- The paper introduces actor and action video segmentation from a sentence.
- Its encoder-decoder architecture performs pixel-level segmentation while explicitly modeling video’s spatio-temporal nature.
- Experiments show feasibility and robustness, adaptation to semantic segmentation of actor-action pairs, and state-of-the-art performance.
Sentence
The supplementary material reports annotation statistics for A2D Sentences and J-HMDB Sentences, presents additional segmentation results, and compares video-based and image-based models.
- Section S1 reports annotation statistics for the A2D Sentences and J-HMDB Sentences datasets.
- Section S2 presents additional segmentation results from the proposed model.
- Section S3 qualitatively compares the video-based model with image-based models from Hu et al. and Li et al.
S1. Dataset statistics
The sentence annotations provide finer-grained actor and action descriptions than the regular datasets’ labels. They also represent multiple linguistic descriptions of the same action.
- Sentence annotations distinguish fine-grained actors within the same super-category, such as man, woman, girl, and boy.
- Natural language represents jumping through variants such as jumping, bouncing, and falling.
- Natural language represents rolling through flipping, turning, rolling, and rotating, and running through moving, running, and chasing.
S2. Segmentation results on A2D Sentences
The supplementary visualizations show sentence-guided segmentation across videos containing deformable and non-deformable objects, including examples with reflecting surfaces and repeated actor instances.
- The model segments deformable objects, including the woman example, and non-deformable objects, including the ball example.
- The ball example demonstrates that the model can handle reflecting surfaces.
- The third video demonstrates the model’s ability to distinguish instances among the same actor and action.
S3. Baseline comparison on A2D Sentences
Qualitative comparisons show that the proposed model segments small actors, entire objects, and fine-grained actor types more effectively than image-based baselines.
- The model segments relatively small actors that both image-based baselines struggle to capture.
- The model achieves better segmentation accuracy than the baseline models in qualitative examples.
- For a car example, the model segments the whole car, whereas both baselines segment only parts.
- The model better distinguishes between different types of actors than the baseline models.