Source-linked AI summary
Collaborative Spatial-Temporal Modeling for Language-Queried Video Actor Segmentation
Tianrui Hui, Shaofei Huang, Si Liu, Zihan Ding, Guanbin Li, Wenguan Wang, Jizhong Han, Fei Wang
TL;DR
Language-queried video actor segmentation must identify the actor performing a described action and produce a precise mask in the target frame, a challenge for approaches that mix spatial information across frames. The paper combines a 3D temporal encoder, a 2D spatial encoder, and language-guided feature interaction; it reports better results on two benchmarks with lower computational overhead.
Problem
Prior 3D-CNN approaches mix spatial information from adjacent frames, while the task requires recognizing the queried actor and producing an accurate target-frame mask.
Source: §1 · p. 2Method
A 3D temporal encoder recognizes queried actions, a 2D spatial encoder extracts target-frame features, and LGFS and CMAM provide language-guided feature fusion and interaction.
Source: §1 · p. 2Results
On two popular benchmarks, the method outperforms previous methods by large margins with 3× less computational overhead.
Source: §5 · p. 8Takeaways & Limitations
The paper concludes that its collaborative framework alleviates the spatial-misalignment issue associated with prior 3D CNNs.
Source: §5 · p. 8The reported setting concerns masks on unseen samples.
Source: §4.3 · p. 6
Abstract
from arXiv · showhide
Language-queried video actor segmentation aims to predict the pixel-level mask of the actor which performs the actions described by a natural language query in the target frames. Existing methods adopt 3D CNNs over the video clip as a general encoder to extract a mixed spatio-temporal feature for the target frame. Though 3D convolutions are amenable to recognizing which actor is performing the queried actions, it also inevitably introduces misaligned spatial information from adjacent frames, which confuses features of the target frame and yields inaccurate segmentation. Therefore, we propose a collaborative spatial-temporal encoder-decoder framework which contains a 3D temporal encoder over the video clip to recognize the queried actions, and a 2D spatial encoder over the target frame to accurately segment the queried actors. In the decoder, a Language-Guided Feature Selection (LGFS) module is proposed to flexibly integrate spatial and temporal features from the two encoders. We also propose a Cross-Modal Adaptive Modulation (CMAM) module to dynamically recombine spatial- and temporal-relevant linguistic features for multimodal feature interaction in each stage of the two encoders. Our method achieves new state-of-the-art performance on two popular benchmarks with less computational overhead than previous approaches.
1. Introduction
Language-queried video actor segmentation predicts pixel masks for actors described by natural-language action queries, while prior 3D-CNN approaches mix spatial information across frames. The paper proposes collaborative spatial-temporal modeling with language-guided feature interaction and reports improved benchmark performance with lower computational overhead.
1. Introduction
- Pixel-level segmentation requires temporal context to distinguish actors performing queried actions and target-frame spatial features to preserve precise masks.The passage illustrates the challenge with two similar cats whose poses and locations vary across frames.Source: §1 · pp. 1–2
- Prior methods use only temporal modeling, so the proposed framework pairs a 3D action-recognizing encoder with a 2D encoder for precise target-frame features.LGFS uses language-generated channel-selection weights to flexibly combine the two encoders' features in the decoder.Source: §1 · p. 2
- CMAM dynamically recombines language features so spatial-relevant words modulate spatial features and temporal-relevant words modulate temporal features.It is inserted at each stage of both encoders for hierarchical visual-linguistic interaction.Source: §1 · p. 2
- The method outperforms previous state-of-the-art methods on two popular benchmarks with less computational overhead.The introduction identifies extensive experiments as supporting this comparison.Source: §1 · p. 2
2. Related Work
Related work covers actor-action segmentation, language-based video localization, and spatio-temporal video modeling. The paper distinguishes its mask-based task and collaborative spatial-temporal design from those approaches.
2.1. Actor and Action Video Segmentation
- Actor-action segmentation progressed from A2D pixel labels and graphical or ranking models to deep detection-segmentation and language-queried methods using dynamic filters, co-attention, capsules, or positional encoding.The paper contrasts these methods, which use mixed spatio-temporal features, with its 2D spatial encoder alongside a 3D temporal encoder.Source: §2.1 · pp. 2–3
2.2. Language-Queried Video Actor Localization
- Language-queried localization methods match sentences to 1D temporal sequences, 2D spatial detections, or 3D tubes; this paper instead identifies actors and actions with segmentation masks.Source: §2.2 · p. 3
2.3. Spatio-Temporal Modeling
- Video spatio-temporal modeling includes 3D CNNs, decomposed (2+1)D convolutions, SlowFast pathways, and temporal shifts; this work combines spatial and temporal encoders.Its architecture applies CMAM across encoder stages and LGFS across decoder stages to interact with and fuse the two feature streams.Source: §2.3 · p. 3, Figure 2 · p. 3
3. Method
The method pairs spatial encoding of the target frame with temporal encoding of the video clip, using language-conditioned interactions in both encoders and a decoder that selectively fuses their features.
3. Method
- The framework uses spatial and temporal visual encoders, CMAM-guided interactions at every encoder stage, and an LGFS decoder that progressively fuses their features into a target-frame mask.Source: §3 · p. 3
3.1. Visual and Linguistic Encoders
- Inception V3 processes the target frame spatially, I3D processes the full clip temporally, and a GRU encodes the query; 8-dimensional coordinates encode pixel relative position.Source: §3.1 · p. 4
3.2. Cross-Modal Adaptive Modulation
- CMAM weights words by visual relevance, turns them into channel-wise gates, and modulates target-frame features; the same operation highlights sentence-relevant features across clip frames.Source: §3.2 · p. 4
3.3. Language-Guided Feature Selection
- LGFS uses pooled sentence features to select spatial-versus-temporal channel weights, combines modulated encoder features at each decoder stage, and progressively upsamples toward mask resolution.Source: §3 · p. 3, §3.3 · p. 5
4. Experiments
On A2D Sentences, the method outperforms prior state-of-the-art methods, and it also gains on J-HMDB when evaluated without finetuning. Ablations, overhead analysis, and qualitative examples examine the framework’s components, efficiency, and predictions.
4.1. Datasets and Evaluation Metrics
- Overall IoU, Mean IoU, Precision@X, and AP evaluate performance on A2D Sentences and J-HMDB Sentences.Precision@X measures the percentage of test samples above an IoU threshold, and AP is computed over [0.50 : 0.05 : 0.95].Source: §4.1 · p. 5
4.2. Implementation Details
- The model samples eight RGB frames with the annotated target frame in the middle and trains encoders and decoders in two stages.The second stage combines pretrained encoders with a randomly initialized decoder and finetunes the decoder while the encoders are fixed.Source: §4.2 · p. 5
4.3. Comparison with State-of-the-arts
- On A2D Sentences, Mean IoU improves by 3.0% over CMDy and 3.2% over PRPE; the method also gains on J-HMDB without finetuning.The J-HMDB evaluation reports approximate zero performance on P@0.9 for all methods, likely because models not trained there cannot produce particularly fine masks on unseen samples.Source: §4.3 · pp. 5–6, Table 1 · p. 6, Table 2 · p. 6
4.4. Ablation Studies
- Spatial-only and temporal-only baselines favor P@0.9 and P@0.5 respectively; combining the encoders boosts all metrics, and LGFS improves performance further.The ablation compares models trained on the A2D Sentences train split and evaluated on its test split.Source: §4.4 · p. 6, Table 3 · p. 7
- Without CMAM, LGFS outperforms elementwise addition and maximization, which produce similar results, supporting language-guided spatial-temporal feature selection.The fusion comparison is conducted without CMAM.Source: §4.4 · pp. 6–7
- CMAM insertion improves segmentation in shallow stages, with relatively significant gains from stages five and four; tested backbone variants retain stable, high performance.The authors suggest deeper features’ higher-level semantics may benefit segmentation.Source: §4.4 · pp. 6–7
4.5. Computational Overhead
- 3× less GFLOPs than CMDy and ACGA are reported for our method with a much smaller input size; the spatial encoder accounts for 9.2% of the full framework’s computational overhead.CMDy’s overhead is estimated from its paper because its code was not released.Source: Table 4 · p. 7, §4.5 · p. 7
4.6. Qualitative Analysis
- Qualitative examples show spatial-only predictions can include irrelevant actors, temporal-only masks can miss local detail, and using both encoders yields precise segmentation.CMAM attention maps associate appearance words with body regions in the spatial encoder and action words with moving body parts in the temporal encoder.Source: §4.6 · pp. 7–8, Figure 4 · p. 8, Figure 5 · p. 8
5. Conclusion and Future Work
The paper concludes with a collaborative encoder-decoder framework that combines temporal action recognition and spatial actor segmentation. It reports stronger performance on two benchmarks with lower computational overhead and identifies lighter encoders as future work.
5. Conclusion and Future Work
- The framework combines a 3D temporal encoder for queried-action recognition with a 2D spatial encoder to segment actors and alleviate spatial misalignment from prior 3D CNNs.Source: §5 · p. 8
- LGFS flexibly fuses spatial-temporal features in the decoder, while CMAM recombines linguistic features for adaptive multimodal interaction in each encoder.Source: §5 · p. 8
- The method outperforms previous methods by large margins on two popular benchmarks with 3× less computational overhead.Source: §5 · p. 8
- Future work aims to accelerate the framework by designing lighter spatial and temporal encoders.Source: §5 · p. 8