PaperlayerView exact revision on arXiv

Source-linked AI summary

CenterCLIP: Token Clustering for Efficient Text-Video Retrieval

Shuai Zhao, Linchao Zhu, Xiaohan Wang, Yi Yang

arXiv:2205.00823v1cs.CVcs.IR

TL;DR

Video transformers generate many redundant tokens from consecutive, similar frames, increasing computation costs for text-video retrieval. CenterCLIP clusters tokens within temporal segments, retains representative centers, and applies attention to the resulting sequence. It achieves state-of-the-art performance while reducing memory and inference costs, with best-case reductions of 35% and 14%, respectively.

  • Problem

    Redundant tokens from consecutive video frames increase computation costs in CLIP-based text-video retrieval.

  • Method

    CenterCLIP independently clusters tokens within temporal segments, retains cluster centers, and uses k-medoids or spectral clustering to form a shorter sequence.

  • Results

    CenterCLIP achieves state-of-the-art performance on four benchmarks, with 35% lower memory cost and 14% faster inference on ActivityNet in the best case.

  • Takeaways & Limitations

    Token clustering reduces redundant computation while enabling segment-level alignment between video and text representations.

  • Takeaways & Limitations

    Clustering outcomes can depend on normalization, distance metrics, and graph construction, which the paper does not explore further.

Abstract

from arXiv · show

Recently, large-scale pre-training methods like CLIP have made great progress in multi-modal research such as text-video retrieval. In CLIP, transformers are vital for modeling complex multi-modal relations. However, in the vision transformer of CLIP, the essential visual tokenization process, which produces discrete visual token sequences, generates many homogeneous tokens due to the redundancy nature of consecutive and similar frames in videos. This significantly increases computation costs and hinders the deployment of video retrieval models in web applications. In this paper, to reduce the number of redundant video tokens, we design a multi-segment token clustering algorithm to find the most representative tokens and drop the non-essential ones. As the frame redundancy occurs mostly in consecutive frames, we divide videos into multiple segments and conduct segment-level clustering. Center tokens from each segment are later concatenated into a new sequence, while their original spatial-temporal relations are well maintained. We instantiate two clustering algorithms to efficiently find deterministic medoids and iteratively partition groups in high dimensional space. Through this token clustering and center selection procedure, we successfully reduce computation costs by removing redundant visual tokens. This method further enhances segment-level semantic alignment between video and text representations, enforcing the spatio-temporal interactions of tokens from within-segment frames. Our method, coined as CenterCLIP, surpasses existing state-of-the-art by a large margin on typical text-video benchmarks, while reducing the training memory cost by 35\% and accelerating the inference speed by 14\% at the best case. The code is available at \href{https://github.com/mzhaoshuai/CenterCLIP}{https://github.com/mzhaoshuai/CenterCLIP}.

1 INTRODUCTION

CenterCLIP addresses redundant visual tokens in video transformers by clustering tokens within temporal segments, retaining representative centers for more efficient retrieval and segment-level alignment. It uses two clustering strategies and reports state-of-the-art results with substantial efficiency gains.

  • Motivation: Video tokenization produces many homogeneous tokens across continuously changing frames, causing repeated computation in CLIP-based retrieval.Similar token embeddings form tight clusters in high-dimensional space.
  • Method: CenterCLIP divides videos into temporal segments, clusters tokens independently within each segment, and drops non-center tokens.The retained center tokens are concatenated into a shorter sequence while preserving segment structure.
  • Method: The method instantiates deterministic k-medoids with KKZ initialization and spectral clustering for high-dimensional token groups.KKZ initialization makes clustering results consistent across runs and can accelerate convergence.
  • Results: CenterCLIP achieves state-of-the-art performance on MSR-VTT, MSVD, LSMDC, and ActivityNet while improving retrieval metrics over the baseline.The reported gains occur across all four benchmarks.
  • Results: 35% lower memory cost and 14% faster inference are reported on ActivityNet compared with the baseline.These are the best-case efficiency figures reported for ActivityNet.

2 RELATED WORKS

The related-work context connects CenterCLIP to large-scale contrastive vision-language pre-training and frames its framework as segment-aware token clustering for video retrieval.

  • Contrastive Vision-Language Pre-Training: Large-scale image-text pre-training has driven progress in visual representation learning and related text-video understanding methods.The text-video pre-training scale is described as smaller than language-image pre-training.
  • CenterCLIP framework: CenterCLIP clusters visual tokens within temporal segments, then uses attention over selected centers to learn segment-level video semantics.The framework example divides a video into three segments of three frames and concatenates centers from each segment.

3 METHODS

CenterCLIP formulates text-video retrieval with normalized text and video encoders, then reduces redundant visual tokens through segment-level clustering while preserving spatial-temporal organization. The method supports segment-level alignment and reduces computation by shortening later transformer sequences.

  • Retrieval framework: Text-video retrieval ranks videos for text queries or texts for video queries using similarity scores from normalized text and video features.The model uses text encoder g and video encoder h, with end-to-end contrastive training over matched and mismatched pairs.
  • Video-token redundancy: Video tokenization produces L|v_i| visual tokens, exceeding 1,000 for many long ActivityNet videos and increasing computation costs.For 224 × 224 inputs, L is 49 for ViT-B/32 and 196 for ViT-B/16.
  • Multi-segment token clustering: Clustering across neighboring frames lets image patches interact through self-attention while preserving segment-level temporal variation and semantic alignment.This design addresses the redundancy of similar neighboring frames without relying only on frame-feature averaging or late fusion.
  • Multi-segment token clustering: CenterCLIP divides videos into temporal segments, clusters tokens independently within each segment, retains cluster centers, and drops non-center tokens.The retained centers are concatenated in their original relative spatial-temporal order.
  • Efficiency: After clustering at the B-th transformer block, subsequent sequence length becomes K rather than the larger original token count, reducing computation when K is much smaller.The clustering module can be inserted at different ViT locations, with earlier insertion reducing more computation.

4.1 Experimental Details

The experiments evaluate CenterCLIP on four text-video retrieval datasets using standardized video preprocessing, sparse frame sampling, and segment-level token clustering. They report retrieval performance alongside GPU memory and inference speed, with clustering settings specified for reproducibility.

  • Datasets and preprocessing: The evaluation uses MSR-VTT, MSVD, LSMDC, and ActivityNet, with videos resized to a 224-pixel shorter side and sampled at 3 fps.MSR-VTT includes 10,000 videos and 200,000 captions; training-7K and training-9K splits are used.
  • Evaluation protocol: Memory is measured as average GPU cost during training, while speed is inference time per video during evaluation on a Tesla V100 GPU.The MSVD, MSR-VTT, and LSMDC tables use 2 and 8 GPUs for ViT-B/32 and ViT-B/16, respectively; ActivityNet uses 8 and 32 GPUs.
  • Optimization and batch configuration: The learning setup uses warmup with cosine decay, AdamW with weight decay 0.2, and dataset-dependent sequence and frame-length limits.ActivityNet uses text length 77 and frame length 60; other datasets use text length 32 and frame length 12.
  • Frame sampling: Training samples one frame per temporal segment, while evaluation uniformly samples 60 frames for ActivityNet and 12 for other datasets.The sampled frames are further divided into S segments for token clustering.
  • Clustering settings: Spectral clustering uses a Gaussian-similarity KNN graph, with neighborhood size tied to segment frame count and Gaussian variance set to 2.0.Token embeddings are not normalized before clustering, and baselines use the same setting.

4.2 Results on Common Benchmarks

CenterCLIP achieves state-of-the-art retrieval performance across four common benchmarks while reducing memory use and, in several settings, accelerating evaluation. Its recall gains are reported across model variants, and spectral clustering and k-medoids++ perform similarly in most cases.

  • Benchmark performance and efficiency: CenterCLIP achieves state-of-the-art performance on MSR-VTT, MSVD, LSMDC, and ActivityNet while reducing memory cost and accelerating evaluation in some settings.The reported best-case efficiency gains include 32% lower memory and 6% faster evaluation on three datasets, and 35% lower memory and 14% faster evaluation on ActivityNet.
  • Efficiency results: 35% lower memory cost and 14% faster evaluation are achieved on ActivityNet in the best case with CenterCLIP ViT-B/32.For MSR-VTT, MSVD, and LSMDC, the best-case ViT-B/32 reduction is 32% memory and the speedup is 6%.
  • Recall improvements: CenterCLIP improves text→video R@1 over the baseline by 1.7% on MSVD, 1.2% on MSR-VTT training-9K, 1.8% on LSMDC, and 2.1% on ActivityNet with ViT-B/32.With ViT-B/16, reported gains vary across datasets and retrieval directions, including larger video→text improvements in some cases.
  • Clustering comparison: Spectral clustering and k-medoids++ achieve similar performance in most cases, while spectral clustering is better on selected R@5 and R@10 metrics and MSVD video→text results.The authors suggest this similarity may reflect nearly spherical token-embedding clusters in high-dimensional space.

4.3 Diagnostic Experiments

Diagnostic experiments examine token-selection baselines, clustering placement and settings, input-frame counts, training choices, and center-token visualizations. They identify trade-offs between retrieval quality, information retention, memory, and computation.

  • More baselines: CenterCLIP outperforms sparse token sampling, while token shifting improves short-video performance without reducing computation costs.The comparison includes pooling, sparse sampling, temporal shift, and token shift baselines.
  • Clustering location: A clustering location at B=6 provides the reported trade-off between memory cost and performance.Smaller B lowers memory cost, but performance decreases when B is decreased or increased from this setting.
  • Cluster number and segment number: Increasing K improves performance while increasing computation, whereas very small S can discard more information and make clustering expensive; moderate S is usually adopted.The cost of a small S is illustrated by segments containing 12 × 49 or 10 × 49 tokens in the examined datasets.
  • Input frames: More input frames increase computation, while too few frames worsen performance; with fixed S, they also enlarge each segment’s clustering workload.The experiments vary N_in on ActivityNet and report the same computation trade-off for clustering.
  • Normalization: Embedding normalization has a trivial effect in the reported comparison, while clustering graph construction and related factors remain unexplored.The paper explicitly states that these factors are outside its focus.
  • Training settings: On ActivityNet, larger learning rates and more training epochs improve the reported baseline results, possibly because long videos contain more varied frames.The authors use mixed precision for short-video datasets and attribute the ActivityNet behavior as a possibility rather than a confirmed mechanism.
  • Center-token visualization: Center tokens retain representative foreground content while dropping much of the similar background, supporting the use of a few typical tokens for discriminative video features.The visualization varies the number of frames within a temporal segment.

5 CONCLUSION

CenterCLIP reduces redundant tokens from continuous video frames through multi-segment clustering and center-token selection. It achieves state-of-the-art performance across four datasets while reducing memory costs and speeding inference.

  • Conclusion: CenterCLIP reduces redundant tokens from continuous video frames and promotes segment-level alignment between video and text representations.The method is built on large-scale image-text pre-trained CLIP representations.
  • Conclusion: CenterCLIP achieves state-of-the-art performance on MSR-VTT, MSVD, LSMDC, and ActivityNet while reducing memory costs and speeding inference.The conclusion reports both retrieval performance and efficiency gains across the four evaluated datasets.
Loading 2205.00823v1…