Actor and Action Video Segmentation from a Sentence

• The paper defines actor and action segmentation from a sentence as pixel-level video segmentation guided by a natural-language input.Source: §1 · p. 1, Figure 1 · p. 1
• Unlike prior methods using fixed actor-action pairs, the approach accepts natural-language descriptions to specify the actor and action.Source: §1 · p. 1
• The proposed model is a fully convolutional, end-to-end trainable encoder-decoder architecture optimized for video.Source: §1 · pp. 1–2
• The authors extend A2D and J-HMDB with more than 7,500 textual sentences describing actors and actions in video content.Source: §1 · p. 2
• Experiments assess sentence-guided segmentation quality, model generalization, and performance on traditional actor-action segmentation.Source: §1 · p. 2

• The A2D dataset and related methods assign video regions labels from a fixed vocabulary of 43 actor-action pairs.Source: §2.1 · p. 2
• The proposed model represents the joint actor-action space with an open set of labels as rich as language.Source: §2.1 · p. 2
• Language-guided segmentation can distinguish fine-grained actors within a super-category and identify different actor and action instances.Source: §2.1 · p. 2
• Pre-trained word embeddings allow inference from words outside the actor-action vocabulary when those words exist in the embedding.Source: §2.1 · p. 2
• Compared with sentence-based person retrieval and localization, this work targets any actor and outputs pixel-wise segmentation of both actor and action.Source: §2.2 · p. 2
• Unlike sentence-based temporal action retrieval or zero-shot tube localization, the paper studies spatio-temporal segmentation from language.Source: §2.3 · p. 2

• The architecture combines a textual encoder, a 3D video encoder, and a decoder for pixel-wise segmentation.Source: §3 · pp. 2–3, Figure 2 · p. 3

Textual encoder

• The textual encoder uses pre-trained word2vec embeddings and a 1D convolution with ReLU and max-pooling to represent the sentence.Source: §3.1 · p. 3

Video encoder

• The video encoder uses 3D convolutional features to preserve spatial information while modeling temporal video dynamics.Source: §3.2 · p. 3

Dynamic filtering

• Dynamic filters generated from the sentence representation are convolved with video representations at multiple resolutions to produce segmentation response maps.Source: §3.3 · p. 4

Decoder

• A deconvolutional network operates directly on the video representation to produce full-resolution masks, better handling small objects and smoother predictions.Source: §3.3 · p. 4

Training

• Training uses binary masks at resolutions 32, 128, and 512, combining their weighted losses to improve gradient flow.Source: §3.4 · p. 4

A2D Sentences

• A2D contains 3,782 videos with pixel-level actor and action labels spanning eight actions and seven actor classes.The dataset is split into 3,036 training videos and 746 testing videos.Source: §4.1 · p. 4
• 6,656 A2D sentences use discriminative expressions to identify actor instances and include 811 nouns, 225 verbs, and 189 adjectives.These annotations refine actor-action pairs with more granular descriptions, such as man, woman, or player for adult.Source: §4.1 · p. 5
• A2D sentences average 7.3 words, or 4.5 content words without function words, compared with 4.7 and 3.6 in ReferIt.The richer descriptions reflect the inclusion of varied verbs, which referring-expression datasets often omit.Source: §4.1 · p. 5

J-HMDB Sentences

• J-HMDB provides 928 clips covering 21 actions, and the paper adds one natural-language target description to each video.The J-HMDB annotations contain 158 nouns, 53 verbs, and 23 adjectives and follow the A2D annotation protocol.Source: §4.2 · p. 5

Dataset release

• The authors provide sentence-annotated examples for both datasets and state that the annotations and model code will be released.Examples appear in Figure 3, with additional dataset details in the supplemental material.Source: §4.2 · p. 5

Ablation Study

• Using 16 input frames raises overall IoU from 48.2% with one frame to 53.6%, highlighting the value of temporal context.The evaluated settings are N = 1, 4, 8, 16, with overall IoU values of 48.2%, 52.2%, 52.8%, and 53.6%, respectively.Source: §5.1 · p. 5
• The convolutional textual encoder reaches 53.6% overall IoU, compared with 51.8% for LSTM and 52.1% for bidirectional LSTM.The model therefore uses the convolutional encoder in subsequent experiments.Source: §5.1 · pp. 5–6
• Multi-resolution loss achieves 53.6% overall IoU versus 49.4% when only the highest resolution is used.The compared resolutions are 32, 128, and 512, with all three contributing in the multi-resolution setting.Source: §5.1 · p. 6
• RGB-flow fusion with RGB weighted twice as strongly as Flow reaches 55.1% overall IoU, exceeding RGB-only performance at 53.6%.RGB alone scores 53.6%, while Flow alone scores 49.5%, using 16 frames for both streams.Source: §5.1 · p. 6

Segmentation from a sentence

• On A2D Sentences, the model outperforms both baselines across all metrics, with RGB inputs improving mAP by 3.5%, overall IoU by 2.1%, and mean IoU by 6.7%.Fusion of RGB and Flow further improves results, and the larger mean-IoU gain indicates stronger performance on small objects.Source: §5.2 · p. 6, Table 1 · p. 6
• The model generalizes from A2D Sentences to J-HMDB Sentences without additional fine-tuning and produces more accurate segmentations at higher overlap values.J-HMDB Sentences contains human actions, with four of its 21 actions overlapping those in A2D Sentences.Source: §5.2 · p. 7, Table 2 · p. 8

Segmentation from actor and action pairs

• On A2D, the method improves state of the art by 4.9% in class-average accuracy and 5.1% in Mean IoU for joint actor-action segmentation.For action segmentation, it improves class-average accuracy by 8.8% and Mean IoU by 7.2%.Source: §5.3 · p. 8

• The paper introduces actor and action video segmentation from a sentence.Source: §6 · p. 8
• Its encoder-decoder architecture performs pixel-level segmentation while explicitly modeling video’s spatio-temporal nature.Source: §6 · p. 8
• Experiments show feasibility and robustness, adaptation to semantic segmentation of actor-action pairs, and state-of-the-art performance.Source: §6 · p. 9

• Section S1 reports annotation statistics for the A2D Sentences and J-HMDB Sentences datasets.Source: Sentence · p. 10
• Section S2 presents additional segmentation results from the proposed model.Source: Sentence · p. 10
• Section S3 qualitatively compares the video-based model with image-based models from Hu et al. and Li et al.Source: Sentence · p. 10

• Sentence annotations distinguish fine-grained actors within the same super-category, such as man, woman, girl, and boy.Source: S1. Dataset statistics · p. 10
• Natural language represents jumping through variants such as jumping, bouncing, and falling.Source: S1. Dataset statistics · p. 10
• Natural language represents rolling through flipping, turning, rolling, and rotating, and running through moving, running, and chasing.Source: S1. Dataset statistics · p. 10

• The model segments deformable objects, including the woman example, and non-deformable objects, including the ball example.Source: S2. Segmentation results on A2D… · p. 10
• The ball example demonstrates that the model can handle reflecting surfaces.Source: S2. Segmentation results on A2D… · p. 10
• The third video demonstrates the model’s ability to distinguish instances among the same actor and action.Source: S2. Segmentation results on A2D… · p. 10

• The model segments relatively small actors that both image-based baselines struggle to capture.Source: S3. Baseline comparison on A2D Sentences · p. 10
• The model achieves better segmentation accuracy than the baseline models in qualitative examples.Source: S3. Baseline comparison on A2D Sentences · p. 10
• For a car example, the model segments the whole car, whereas both baselines segment only parts.Source: S3. Baseline comparison on A2D Sentences · p. 10
• The model better distinguishes between different types of actors than the baseline models.Source: S3. Baseline comparison on A2D Sentences · p. 10