Source-linked AI summary
TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval
Yuqi Liu, Pengfei Xiong, Luhui Xu, Shengming Cao, Qin Jin
TL;DR
Text-video retrieval needs video representations that preserve subtle movements and salient spatial information. TS2-Net addresses this with token shifting and selection, and outperforms state-of-the-art methods on five retrieval benchmarks.
Problem
Video representation must model subtle movements and local spatial semantics without losing important objects or allowing irrelevant background tokens to dominate aggregation.
Method
TS2-Net combines a parameter-free token shift transformer for temporal modeling with a token selection transformer that selects top-K informative tokens per frame.
Results
TS2-Net outperforms state-of-the-art methods on MSR-VTT, VATEX, LSMDC, ActivityNet-Caption, and DiDeMo.
Takeaways & Limitations
The approach provides a video encoder that models subtle movements and salient object semantics through token shift and selection modules.
Takeaways & Limitations
Applying token shift in shallow layers or across too many layers can hurt spatial modeling and retrieval performance.
Abstract
from arXiv · showhide
Text-Video retrieval is a task of great practical value and has received increasing attention, among which learning spatial-temporal video representation is one of the research hotspots. The video encoders in the state-of-the-art video retrieval models usually directly adopt the pre-trained vision backbones with the network structure fixed, they therefore can not be further improved to produce the fine-grained spatial-temporal video representation. In this paper, we propose Token Shift and Selection Network (TS2-Net), a novel token shift and selection transformer architecture, which dynamically adjusts the token sequence and selects informative tokens in both temporal and spatial dimensions from input video samples. The token shift module temporally shifts the whole token features back-and-forth across adjacent frames, to preserve the complete token representation and capture subtle movements. Then the token selection module selects tokens that contribute most to local spatial semantics. Based on thorough experiments, the proposed TS2-Net achieves state-of-the-art performance on major text-video retrieval benchmarks, including new records on MSRVTT, VATEX, LSMDC, ActivityNet, and DiDeMo.
1 Introduction
Text-video retrieval requires video representations that capture both spatial details and temporal motion. TS2-Net addresses this with token shift and token selection modules, achieving state-of-the-art results across major benchmarks.
- Video retrieval systems map text queries and candidate videos into a shared embedding space for similarity matching.
- Fine-grained representation must capture both motion between frames and entities within frames.
- Existing transformer approaches motivate patch enhancement to model small objects and subtle movements more effectively.
- TS2-Net shifts whole token features across adjacent frames and selects top-K informative tokens for temporal motion and salient spatial semantics.The shift operation preserves complete token features, while selection emphasizes locally informative tokens.
- TS2-Net achieves state-of-the-art performance on MSRVTT, VATEX, LSMDC, ActivityNet-Caption, and DiDeMo.Ablation studies report benefits from both token shift and token selection modules.
2 Related Work
Prior text-video retrieval methods use feature extractors, fusion modules, or end-to-end pretrained models, but often lack fine-grained spatial or temporal representation. Related video transformers and shift-based methods provide the architectural context for TS2-Net.
- Earlier retrieval systems commonly combine offline feature extractors with feature-fusion modules.
- End-to-end methods jointly train semantic alignment networks with low-level feature extractors or transfer pretrained CLIP knowledge.
- Existing models still lack fine-grained representation capacity in either the spatial or temporal dimension.
- ViT-based video models such as TimeSformer and ViViT model spatial-temporal video structure, while TSM and Shift Transformer explore temporal shift operations.
3 Method
TS2-Net encodes videos with token shift and token selection transformers before matching frame-aware video representations with text. Token shifting models local temporal motion, while token selection retains informative spatial tokens for global spatial-temporal aggregation.
- Overall Architecture: TS2-Net comprises text encoding, video encoding, and text-video matching, with Token Shift and Token Selection Transformers in the video encoder.The text encoder produces query representation q, while the video encoder produces frame-wise representations v = {f1, f2, ..., ft}.
- Token Shift Transformer: Token shift moves whole spatial token features between adjacent frames to capture subtle local movements without compromising each video token’s integrity.The operation is parameter-free and implemented bidirectionally; residual connections preserve original features, and shifting is applied only in deeper layers.
- Token Selection Transformer: Token selection addresses aggregation loss and redundancy by selecting top-K informative tokens from each frame, especially tokens containing salient object semantics.Using only [CLS] can miss spatial information, whereas aggregating all tokens can allow irrelevant background tokens to dominate the video representation.
- Token Selection Transformer: A reduced token representation is scored by an MLP and Softmax, after which an indicator matrix extracts the K tokens with the highest importance scores.The selected tokens from all frames enter a joint spatial-temporal transformer, and the most informative token from each frame provides frame-wise video encoding.
- Differentiable TopK: The differentiable top-K selection uses a perturbed maximum method to optimize the indicator matrix while retaining the top-K operation.The method formulates selection as maximizing the inner product between importance scores S and an indicator matrix M over a constraint set.
- Text-Video Matching: Frame-level text-video similarity uses cosine similarity between query q and frame representation fi, while the final score is a weighted combination of frame-level similarities.During inference, videos are ranked by the matching score; training uses symmetric cross-entropy loss with a trainable scaling parameter τ.
4 Experiment
Experiments evaluate TS2-Net through ablations, benchmark comparisons, and qualitative retrieval examples. Results support gains from token shift and token selection for fine-grained motion and object modeling across multiple datasets.
- Experimental settings: Evaluations cover five text-video benchmarks using R@K, median rank, mean rank, and rsum metrics.The benchmarks are MSR-VTT, VATEX, LSMDC, ActivityNet-Caption, and DiDeMo; R@K uses K=1,5,10.
- Ablation experiments: The baseline for ablation removes both token shift and token selection modules from TS2-Net on the MSR-VTT 1k-A split.Ablations separately examine the modules and their parameter settings.
- Token shift ablation: Shifting tokens in deeper layers improves retrieval, whereas shifting shallow layers damages spatial modeling; a 25% shift ratio performs best.The selected configuration inserts token shift in layers 11–12 and shifts 25% of tokens.
- Token shift ablation: The proposed token shift outperforms alternative shift variants because it preserves complete token features and better captures subtle movement such as “shake hand.”The comparison is reported on the MSR-VTT-1k-A test split.
- Benchmark comparisons: TS2-Net outperforms previous methods on MSR-VTT and shows consistent improvements across VATEX, LSMDC, ActivityNet-Caption, and DiDeMo.The authors attribute the gains to jointly encoding spatial and temporal features with token shift and token selection.
- Qualitative results: On a 103-query subset emphasizing small objects and slight movements, TS2-Net achieves 79.6 R@1 versus 39.8 for CLIP4Clip.The subset is drawn from the MSR-VTT-1kA test set and evaluates retrieval against the whole test set.
5 Conclusion
The conclusion presents TS2-Net as a transformer architecture that improves video encoding through token shift and selection. It reports state-of-the-art performance across five text-video retrieval benchmarks.
- 5 Conclusion: TS2-Net combines token shift and token selection modules to improve video representations for text-video retrieval.Token shift targets subtle movements, while token selection enhances salient object modeling.
- 5 Conclusion: The authors report state-of-the-art performance on MSR-VTT, VATEX, LSMDC, ActivityNet-Caption, and DiDeMo.
A Inverted Softmax.
The appendix compares TS2-Net with methods using inverted softmax to address hubness during retrieval inference.
- A Inverted Softmax: Hubness describes a data point appearing among the k nearest neighbors of other data points.
- A Inverted Softmax: QB-Norm uses query-bank normalization with dynamic inverted softmax, while other methods report comparisons using basic inverted softmax.
- A Inverted Softmax: TS2-Net surpasses the compared methods under the inverted-softmax comparison reported in Table 6.
B Evaluation Summary on Different Benchmarks
TS2-Net maintains state-of-the-art performance across benchmarks with different caption styles, domains, and video or query lengths. The evaluation is presented as evidence of generalization ability.
- B Evaluation Summary on Different Benchmarks: VATEX and MSR-VTT contain standard captions, LSMDC covers movie-domain clips without movie overlap between training and test sets, and ActivityNet-Caption and DiDeMo use paragraph-video retrieval.
- B Evaluation Summary on Different Benchmarks: TS2-Net consistently maintains state-of-the-art performance across all five benchmark datasets.The datasets have substantially different characteristics and semantic demands.
C More Qualitative Results
Qualitative examples show TS2-Net retrieving videos that match fine-grained actions, objects, and adjectives. Failure cases also reveal disagreements between benchmark ground truth and semantic relevance, including vague annotations.
- TS2-Net distinguishes subtle actions, such as horses having fun versus stationary horses, in qualitative retrieval examples.The examples indicate sensitivity to action differences rather than only broad visual content.
- The model retrieves videos matching small objects and adjective-level details, including thought bubbles, mental bowls, and overweight people.These examples contrast the retrieved videos with visually similar alternatives such as glass bowls or normal people.
- Some failures reflect annotation disagreement: a retrieved video may better match the query than the designated ground-truth video.One example contains both the queried cup and talking, whereas the ground-truth video lacks talking.
- Vague and general query annotations account for 1-2% of the datasets and may limit measured retrieval performance.The authors suggest that more discriminative annotations could improve all metrics.
- Retrieval results are ranked by similarity scores, with green boxes marking ground-truth videos and red boxes marking incorrect retrievals.The visualizations include both additional examples and failure cases.