PaperlayerView exact revision on arXiv

Source-linked AI summary

CenterCLIP: Token Clustering for Efficient Text-Video Retrieval

Shuai Zhao, Linchao Zhu, Xiaohan Wang, Yi Yang

arXiv:2205.00823v1cs.CVcs.IR

TL;DR

CLIP-based video retrieval processes many similar visual tokens from redundant frames, increasing computation. CenterCLIP clusters tokens within temporal segments and reports state-of-the-art performance on four benchmarks, with lower memory costs and faster inference.

  • Problem

    CLIP video encoders produce many similar tokens from consecutive frames, while long videos can yield over 1,000 visual tokens and repeated computation.

    Source: §1 · p. 2, §3.1.2 · p. 4

  • Method

    CenterCLIP clusters visual tokens across frames within temporal segments, retaining cluster-center tokens and dropping non-center tokens to support segment-level video-text alignment.

    Source: §3.2 · p. 4, §5 · p. 10

  • Results

    CenterCLIP achieves state-of-the-art performance on four benchmarks; on ActivityNet, it reduces memory cost by 35% and speeds inference by 14% versus the baseline.

    Source: §1 · p. 2, §4.2 · p. 8

  • Takeaways & Limitations

    The experiments support token clustering as an approach that combines strong retrieval performance with reduced memory costs and faster inference on the evaluated benchmarks.

    Source: §4.2 · p. 8, §5 · p. 10

    The study does not further explore how choices for constructing spectral-clustering graphs affect results.

    Source: §4.3.5 · p. 9

Abstract

from arXiv · show

Recently, large-scale pre-training methods like CLIP have made great progress in multi-modal research such as text-video retrieval. In CLIP, transformers are vital for modeling complex multi-modal relations. However, in the vision transformer of CLIP, the essential visual tokenization process, which produces discrete visual token sequences, generates many homogeneous tokens due to the redundancy nature of consecutive and similar frames in videos. This significantly increases computation costs and hinders the deployment of video retrieval models in web applications. In this paper, to reduce the number of redundant video tokens, we design a multi-segment token clustering algorithm to find the most representative tokens and drop the non-essential ones. As the frame redundancy occurs mostly in consecutive frames, we divide videos into multiple segments and conduct segment-level clustering. Center tokens from each segment are later concatenated into a new sequence, while their original spatial-temporal relations are well maintained. We instantiate two clustering algorithms to efficiently find deterministic medoids and iteratively partition groups in high dimensional space. Through this token clustering and center selection procedure, we successfully reduce computation costs by removing redundant visual tokens. This method further enhances segment-level semantic alignment between video and text representations, enforcing the spatio-temporal interactions of tokens from within-segment frames. Our method, coined as CenterCLIP, surpasses existing state-of-the-art by a large margin on typical text-video benchmarks, while reducing the training memory cost by 35\% and accelerating the inference speed by 14\% at the best case. The code is available at \href{https://github.com/mzhaoshuai/CenterCLIP}{https://github.com/mzhaoshuai/CenterCLIP}.

1 INTRODUCTION

Text-video retrieval is increasingly important, but CLIP-based video processing repeats computation over many similar visual tokens. CenterCLIP clusters tokens within temporal segments, retaining centers in their original spatiotemporal order to reduce redundancy and support segment-level video-text alignment.

1 INTRODUCTION

  • CLIP4clip [32] transfers CLIP [39] knowledge to text-video retrieval and reports more than 30% recall improvement on ActivityNet over prior state of the art.The passage presents this as evidence of the value of billion-scale image-text contrastive pre-training; CLIP uses vision-transformer patch tokens processed by self-attention.Source: §1 · p. 2
  • Consecutive video frames produce many similar visual tokens, whose repeated processing adds unnecessary computation and hinders training and web deployment.The introduction describes token embeddings from different frames forming tight clusters and proposes retaining representative center tokens for visual representation learning.Source: §1 · p. 2
  • CenterCLIP clusters token embeddings independently within equal-length temporal segments, keeps cluster centers, and orders them by their original spatial-temporal positions before transformer processing.Attention among retained tokens from within-segment frames supports segment-level video representations, which are aligned with text through contrastive learning.Source: §1 · p. 2
  • The method instantiates k-medoids with deterministic KKZ initialization for consistent runs and faster convergence, and spectral clustering for high-dimensional data.Source: §1 · p. 2
  • On ActivityNet, CenterCLIP reduces memory cost by 35% and speeds inference by 14% compared to the baseline, while achieving state-of-the-art results on four benchmarks.The benchmarks are MSR-VTT, MSVD, LSMDC [42], and ActivityNet; the passage reports significant retrieval-metric improvements on all four.Source: §1 · p. 2

2 RELATED WORKS

The related work situates CenterCLIP in CLIP-based language-video transfer and efficient transformer research. It distinguishes the method by using image-text-pretrained CLIP and parameter-free segment-level semantic alignment for text-video retrieval.

Contrastive Vision-Language Pre-Training

  • Language-video pre-training is smaller than language-image pre-training because it is more expensive, and CenterCLIP follows CLIP4clip in transferring CLIP knowledge to text-video retrieval.Source: §2 · p. 2

Text-video Retrieval

  • End-to-end pre-training followed by task-specific fine-tuning is increasingly popular for language-video understanding; CenterCLIP instead inherits CLIP trained on image-text pairs rather than video-text pairs.Source: §2 · p. 2

Efficient Transformer

  • Unlike transformer-efficiency methods that alter attention or select important tokens, CenterCLIP is parameter-free and introduces segment-level semantic alignment of text and video representations.It clusters tokens independently within video segments, then concatenates cluster centers into a sequence whose attention supports segment-level video semantics while reducing computation.Source: §2 · p. 2, Figure 2 · p. 3

3 METHODS

CenterCLIP reduces redundant video-token computation by clustering tokens within temporal segments and retaining representative centers. Its CLIP-based retrieval model uses deterministic medoid and spectral clustering methods to support efficient video-text encoding.

3.1 Preliminary

  • The retrieval model pairs CLIP-initialized text and video encoders, normalizes their features, and trains end-to-end with bidirectional normalized-softmax losses over matched in-batch positives and other in-batch negatives [27] [39] [66].The text encoder is a 12-layer, 512-width transformer with 8 attention heads; its final [EOS] representation is normalized and projected into the joint embedding space.Source: §3.1 · p. 3, §3.1.1 · p. 3

3.1.2 Video encoder.

  • ViT tokenization produces L|v_i| visual tokens—49 for ViT-B/32 or 196 for ViT-B/16 at 224 × 224—and long videos can exceed 1,000 tokens, motivating redundancy reduction.The video encoder uses ViT-B/32, which projects non-overlapping frame patches into embeddings independently; each patch is 32 × 32 pixels.Source: §3.1.2 · p. 4

3.2 Multi-segment Token Clustering

  • Segment-wise clustering lets tokens from neighboring frames interact through self-attention, retains cluster centers in their original spatial-temporal order, and drops other tokens for segment-level text-video alignment.Clustering can be inserted at different ViT stages; after clustering into K tokens, subsequent transformer blocks process K tokens instead of the original sequence, reducing computation when L|v_i|/S >> K.Source: §3.2 · p. 4

3.2.1 k-medoids++.

  • To avoid unstable random initialization, k-medoids++ uses deterministic KKZ seeding and nearest-token medoids rather than centroids, avoiding shifts from mean-pooled token embeddings.KKZ selects the maximum-norm point first, then points farthest from existing centroids; the resulting initialization makes clustering deterministic and accelerates convergence.Source: §3.2.1 · pp. 4–5

3.2.2 Spectral clustering.

  • Because high-dimensional token clusters may be non-spherical, the method also uses normalized spectral clustering to partition a similarity graph by within-group and between-group connection weights.It computes Laplacian eigenvectors using SVD and applies a sign-correction algorithm to resolve singular-vector direction ambiguity before clustering the reduced representations.Source: §3.2.2 · p. 5

4 EXPERIMENTS · 4.1 Experimental Details

CenterCLIP is evaluated on four text-video retrieval datasets under dataset-specific sampling and sequence settings. The section defines its clustering configuration, training protocol, and retrieval metrics.

4.1 Experimental Details

  • Experiments cover MSR-VTT, MSVD, LSMDC, and ActivityNet, using MSR-VTT training-9K by default and testing on test 1k-A.MSR-VTT training-7K is also used for baseline comparison; ActivityNet is evaluated with video-paragraph retrieval on val1.Source: §4.1 · pp. 5–6
  • Results are tabulated for MSVD, ActivityNet, MSR-VTT, and LSMDC, with average training GPU memory and evaluation inference time per video reported.The captions specify Tesla V100 evaluation for speed measurements; ActivityNet’s ViT-B/16 baseline runs out of memory on 32 Tesla V100 GPUs.Source: Table 1 · p. 6, Table 2 · p. 6, Table 3 · p. 7
  • Training uses warm-up and cosine learning-rate decay, AdamW [31] with weight decay 0.2, and mixed precision [33].The learning rate is 1e-5 for ActivityNet and 5e-6 for other datasets; maximum epochs are 8 and 5, respectively.Source: §4.1 · p. 6
  • ActivityNet uses maximum text length 77 and 60 frames; other datasets use text length 32 and 12 frames, with total batch size 128.ViT-B/32 ActivityNet experiments use 8 Tesla V100 GPUs; other datasets require at least 2 RTX 3090 GPUs.Source: §4.1 · p. 6
  • CenterCLIP settings use (B_a-S, K): clustering follows transformer block a, divides frames into S temporal segments, and selects K centers.Spectral clustering uses a Gaussian similarity with σ = 2.0 and no embedding normalization; baselines use the same setting.Source: §4.1 · pp. 6, 8
  • Frames are sparsely sampled following TSN [54]: one random frame per segment during training and uniformly sampled frames during evaluation.The sampled frames are further divided into S segments for token clustering; N_in is 60 for ActivityNet and 12 for other datasets.Source: §4.1 · p. 6
  • Retrieval is measured by R@1, R@5, and R@10, plus median rank (MdR) and mean rank (MnR).Higher recall and lower rank indicate better performance.Source: §4.1 · p. 6

4.2 Results on Common Benchmarks

CenterCLIP achieves state-of-the-art performance on all four benchmarks while reducing memory use, and ViT-B/32 also speeds evaluation. Its retrieval gains and comparisons of two clustering algorithms further support the method’s effectiveness.

4.2 Results on Common Benchmarks

  • CenterCLIP achieves state-of-the-art performance on all four datasets, with ViT-B/32 reducing memory cost by 32% and accelerating evaluation by 6% for MSR-VTT, MSVD, and LSMDC.For ActivityNet, ViT-B/32 reduces memory cost by 35% and speeds evaluation by 14% in the best case; ViT-B/16 reduces memory cost by 32% but does not gain speedup.Source: §4.2 · p. 8
  • CenterCLIP improves recall over the baseline, with ViT-B/32 text→video R@1 gains of 1.7% on MSVD, 1.2% on MSR-VTT (training-9K), 1.8% on LSMDC, and 2.1% on ActivityNet.With ViT-B/16, the passage reports text→video R@1 gains of 1.0%, 2.8%, and 0.1%, and video→text R@1 gains of 5.7%, 5.2%, and 2.0%; it notes larger video→text improvements in this case.Source: §4.2 · p. 8
  • Spectral clustering and k-medoids++ perform similarly in most cases, though spectral clustering achieves better R@5 and R@10 on MSR-VTT and ActivityNet and the best video→text results on MSVD.The authors suggest token-embedding clusters may be nearly spherical in high-dimensional space, potentially explaining the similar performance.Source: §4.2 · p. 8

4.3 Diagnostic Experiments

These diagnostics examine token-selection and clustering-design choices in CenterCLIP, showing tradeoffs in retrieval performance, memory, and computation while illustrating which tokens clustering retains.

4.3.1 More baselines.

  • CenterCLIP substantially outperformed sparse token sampling, while token shift performed well on short videos but did not reduce computation costs.Sparse sampling was slightly better than baseline on MSR-VTT and LSMDC; shifting all image patches did not work.Source: §4.3.1 · p. 8, Table 5 · p. 8

4.3.2 The place of performing token clustering.

  • B=6 offered the reported tradeoff between memory cost and performance; memory fell as B decreased, while performance declined as B moved lower or higher.The authors use B=6 as their default setting.Source: §4.3.2 · p. 8, Figure 3 · p. 8
  • Progressive clustering achieves pretty good R@5, R@10, MdR, and memory-cost reduction, but repeated clustering increases time complexity and is unsuitable for large token sets; therefore, the work generally clusters once.Source: §4.3.2 · p. 9

4.3.3 The number of cluster K and segment S.

  • Increasing K improved performance but raised computation costs; small S could lose information and increase clustering cost, so a moderate S was generally adopted.Exceptions to a small S performing better included S=1 on LSMDC and S=6 on ActivityNet, where segments contained 12 × 49 and 10 × 49 tokens, respectively.Source: §4.3.2 · p. 9, Figure 4 · p. 9

4.3.4 The number of input frames Nin.

  • More input frames increased computation costs, while too few frames worsened performance on ActivityNet; with S fixed, more frames also increased clustering costs.The frame-count experiment used CenterCLIP (B6 − 15, 49) on ActivityNet.Source: §4.3.2 · p. 9, Table 7 · p. 9

4.3.5 Normalization of token embeddings.

  • ℓ2 normalization made only a trivial difference to clustering results on LSMDC, with normalization choices corresponding to different distance metrics.The authors did not further explore other clustering factors, such as graph construction in spectral clustering.Source: §4.3.5 · p. 9, Table 8 · p. 9

4.3.6 Learning rate and training epochs.

  • 42.4 T→V R@1 with mixed precision at lr=5e-6 exceeded the 42.1 T→V R@1 CLIP4clip baseline on MSR-VTT test 1k-A.At lr=1e-7, mixed-precision and non-mixed-precision results were 39.7 and 41.7 T→V R@1, respectively; short-video runs used lr=5e-6, while ActivityNet benefited from a larger learning rate and more epochs.Source: §4.3.6 · pp. 9–10, Table 9 · p. 9

4.3.7 Visualization of image patches of center tokens after clustering.

  • Visualizations show clustering retaining representative foreground-animal tokens while discarding some similar-background tokens, supporting the sufficiency of typical tokens for discriminative video features.The examples are shown for different numbers of video frames within a temporal segment.Source: §4.3.7 · p. 10, Figure 5 · p. 10

5 CONCLUSION

CenterCLIP uses multi-segment clustering to reduce redundant video tokens and achieve segment-level video–text alignment. It reports state-of-the-art performance on four datasets while reducing memory costs and speeding inference.

5 CONCLUSION

  • Its multi-segment clustering reduces redundant tokens from continuous video frames and achieves segment-level alignment of video and text representations for retrieval.Source: §5 · p. 10
  • CenterCLIP, based on pre-trained CLIP, keeps cluster-center tokens and drops non-center tokens.Source: §5 · p. 10
  • CenterCLIP achieves state-of-the-art performance on MSR-VTT, MSVD, LSMDC, and ActivityNet, surpassing prior SOTA by a large margin while reducing memory costs and speeding inference.Source: §5 · p. 10

Related papers

Loading 2205.00823v1…