PaperlayerView exact revision on arXiv

Source-linked AI summary

TAME: Temporal-Aware Mixture-of-Experts for Text-Video Retrieval

Uicheol Jung, Juyoung Hong, Hojung Kwon, Yukyung Choi

arXiv:2609.02204v1cs.CV

TL;DR

Text–video retrieval needs temporal modeling beyond CLIP’s frame aggregation because videos contain heterogeneous frames and evolving semantic structure. TAME adds sparse expert routing, cross-frame tokens, and temporal similarity aggregation to CLIP, consistently improving retrieval across TVR benchmarks. The method’s benefits depend on routing stabilization and selective temporal mixing, which introduce practical trade-offs.

  • Problem

    CLIP-based video retrieval often compresses heterogeneous frame representations without modeling temporal dependencies and semantic transitions.

  • Method

    TAME extends CLIP with sparse MoE layers and frame-consistent routing, FT tokens for cross-frame context, and CTIA for sentence-conditioned temporal aggregation.

  • Results

    TAME consistently outperforms CLIP-based baselines across MSR-VTT, DiDeMo, MSVD, LSMDC, and ActivityNet.

  • Takeaways & Limitations

    The results support TAME as a generalizable CLIP-based approach for text–video retrieval across diverse benchmarks.

  • Takeaways & Limitations

    TAME introduces routing-stability and temporal-token overhead, requiring careful load balancing and selective temporal mixing to control memory cost.

Abstract

from arXiv · show

Text-Video Retrieval (TVR) retrieves videos that match a natural-language query, but extending image-text models such as CLIP to videos is fundamentally limited by the lack of temporal modeling. Videos exhibit frame-wise heterogeneity in appearance and motion, and compressing all frames into a single representation often obscures temporal structure and semantic transitions. To address this, we propose Temporal-Aware Mixture-of-Experts for Text-Video Retrieval (TAME), a CLIP-based framework that jointly models frame-level structure and temporal relations. First, we integrate sparse Mixture-of-Experts (MoE) layers into both CLIP encoders and apply frame-consistent routing on the vision branch so that experts specialize according to frame-level visual patterns while preserving the original vision-language alignment. Second, we introduce Frame-Temporal (FT) tokens that aggregate global cross-frame information and feed it back to each frame, enabling the visual encoder to capture long-range temporal dependencies without harming local details. Third, we design a Cross-Temporal Interaction and Aggregation (CTIA) module that refines frame-wise sentence-video similarities through staged temporal filtering and fusion. Experiments on standard TVR benchmarks show that TAME consistently improves over CLIP-based baselines. On MSR-VTT, it improves R@1 by 4.0 over CLIP4Clip, and also achieves consistent gains on DiDeMo, MSVD, LSMDC, and ActivityNet. The code is available at https://github.com/sejong-rcv/TAME.

1. Introduction

TAME addresses temporal and representational limitations in CLIP-based text–video retrieval by combining frame-consistent expert routing, lightweight cross-frame context, and sentence-conditioned temporal aggregation. It retains CLIP alignment while improving frame-aware video matching across benchmarks.

  • TVR requires fine-grained alignment between visual signals and linguistic semantics at the appropriate spatiotemporal granularity.
  • CLIP-based methods commonly mean-pool frame embeddings, overlooking temporal structure, heterogeneous frame patterns, and frame importance.
  • A single dense encoder pathway limits adaptation to heterogeneous visual patterns across static and dynamic scenes, backgrounds, and object compositions.
  • TAME integrates sparse MoE layers into both CLIP encoders with frame-consistent visual routing to expand representational diversity while preserving CLIP alignment.
  • FT tokens propagate compact temporal context across frames while preserving patch-level spatial structure, and CTIA refines frame–sentence similarities into a coherent video-level score.
  • TAME consistently improves over CLIP-based baselines on MSR-VTT, DiDeMo, MSVD, LSMDC, and ActivityNet.

2. Related Work

Related work builds on CLIP dual encoders, temporal modules, and sparse expert architectures for retrieval and efficient representation learning. TAME differs by tailoring routing to text–video retrieval while preserving cross-modal alignment.

  • CLIP-based TVR methods share pretrained dual encoders and similarity heads but differ in frame pooling, sentence–video alignment, and temporal modules.
  • TAME combines sparse MoE, FT tokens, and CTIA as add-on modules to enhance CLIP-based text–video retrieval.
  • Scaling dense models can increase memory, communication, and power costs, motivating sparse approaches with more efficient active computation.
  • Vision and video MoE research routes visual tokens conditionally, while CLIP-MoE and CLIP-UP explore sparse upcycling of pretrained CLIP models.
  • Unlike standard vision and video MoEs, TAME tailors text-token and frame-wise routing to retrieval while maintaining pretrained vision–language alignment.
  • Hash-code retrieval methods in medical imaging form a complementary literature focused on compact representations for efficient indexing.

3. Method

TAME extends CLIP for text-video retrieval by combining sparse expert routing, frame-temporal context, and cross-temporal similarity aggregation. Its method preserves aligned text-video representations while modeling heterogeneous frames and temporal relationships.

  • CLIP Encoder: TAME uses pretrained CLIP dual encoders to produce a global sentence embedding and frame-level video representations for retrieval.The sentence representation is the text [EOS] embedding, while each frame representation is obtained from its visual [CLS] token.
  • MoE-Enhanced CLIP: Sparse MoE layers replace selected dense FFNs in both encoders, using top-K routing over position-wise MLP experts to increase representational capacity.Experts are initialized from the pretrained CLIP FFN, while the router assigns sparse mixtures of expert outputs.
  • MoE-Enhanced CLIP: Frame-wise visual routing applies one expert-selection pattern across all patches in a frame, promoting coherent frame-level specialization and reducing patch-level routing noise.The design addresses inconsistent patch routing by deriving routing from the frame-level [CLS] token and applying the resulting weights uniformly to patch tokens.
  • Frame-Temporal Tokens: Frame–Temporal tokens provide cross-frame context to the vision encoder while retaining patch-level spatial structure.They address the limitation that independently processed frames cannot capture temporal continuity or cross-frame dependencies.
  • Cross-Temporal Interaction & Aggregation (CTIA): CTIA transforms framewise sentence–frame similarities into a temporal aggregation score using sharpening, local Gaussian smoothing, graph propagation, and learnable fusion.The final retrieval score averages the baseline video-level similarity with the CTIA similarity.
  • Training Objective: A batch-hard margin loss further separates each positive text-video pair from its hardest negative during training.For each anchor, the loss compares the positive similarity with the maximum negative similarity under a margin.

4. Experiments

The experiments evaluate TAME on five standard text–video retrieval benchmarks using established splits, retrieval directions, and ranking metrics. Training initializes from CLIP and uses specified optimization, frame, token, and inference settings.

  • Datasets: TAME is evaluated on MSR-VTT, DiDeMo, MSVD, LSMDC, and ActivityNet under standard splits and protocols.MSR-VTT uses 9,000 clips for training and the standard 1k-A test split; DiDeMo uses paragraph-to-video retrieval.
  • Metrics: Evaluation covers both text-to-video and video-to-text retrieval using R@1, R@5, R@10, MdR, and MnR.Recall measures whether the first correct match appears within the top K, while MdR and MnR measure median and mean first-correct-match rank.
  • Implementation: The model is initialized with pretrained CLIP ViT-B/32 weights and optimized with Adam using a cosine learning-rate schedule.The encoders use an initial learning rate of 1 × 10−7, while other modules use 1 × 10−4.
  • Implementation: The reported MSR-VTT results use a maximum token length of 32 and a maximum of 12 frames per video.These settings apply to MSR-VTT, MSVD, and LSMDC.
  • Reporting protocol: Inference-time DSL reweights similarities along query and video dimensions without affecting training.The reweighted scores are combined into the final evaluation score when DSL is enabled.

4.4. Performance Comparisons

TAME improves over CLIP-based baselines on MSR-VTT and generalizes consistently across DiDeMo, MSVD, LSMDC, and ActivityNet. Its gains are reported for standard recall and ranking measures, with DSL providing additional ranking benefits in some settings.

  • MSR-VTT: +4.0 R@1 over CLIP4Clip is achieved by TAME on MSR-VTT.TAME also exceeds X-CLIP by +1.9 R@1 while using video–sentence and frame–sentence alignment.
  • Cross-dataset results: TAME demonstrates stable generalization across DiDeMo, MSVD, LSMDC, and ActivityNet.The reported results describe consistent improvements over the baseline across these datasets.
  • Cross-dataset results: +2.7 R@1 over the baseline is reported on DiDeMo, while MSVD improves by +1.4 R@1.These are text-to-video retrieval results reported in the cross-dataset comparison.
  • Cross-dataset results: +1.5 R@1 over CLIP4Clip is reported on LSMDC, increasing from 22.6 to 24.1.LSMDC MnR also decreases from 61.0 to 57.6.
  • Cross-dataset results: +2.2 R@1 over CLIP4Clip is reported on ActivityNet, improving from 40.5 to 42…
  • DSL analysis: On LSMDC, DSL raises R@5 from 41.4 to 43.3 and R@10 from 51.3 to 51.9 while lowering MnR from 57.6 to 56.0.The DSL result also reports a slight R@1 decrease from 24.1 to 23.4.

4.5. Ablation Study

The ablations show that MoE requires load balancing for stable routing, while temporal components are most effective when CTIA organizes frame-level evidence across time. Upper-layer MoE placement is favored over broader placement.

  • MoE and load balancing: MoE insertion without regularization does not immediately improve retrieval and introduces routing instability across frames and tokens.The authors associate this setting with inconsistent specialization and poorer retrieval alignment.
  • MoE and load balancing: Without LBL, the text MoE at {12} develops strong routing concentration, with top-1 share 0.75 ∼0.80 and normalized entropy ≈0.50 ∼0.56.The same setting has CV ≈1.17 ∼1.28, indicating pronounced imbalance near the top layer.
  • MoE and load balancing: With LBL, routing remains near-uniform at placements {11, 12}, with normalized entropy ≈0.99 ∼1.00 and low top-1 share and CV.The diagnostics support LBL as a stabilizer that mitigates early degradation and router collapse.
  • MoE and load balancing: Row 3 does not yet surpass the dense CLIP4Clip baseline, indicating that LBL stabilizes expert utilization but does not directly boost retrieval alignment on its own.The authors describe LBL as a necessary intermediate step for reliably training the MoE backbone.
  • Temporal modeling: FT tokens alone do not materially improve retrieval because they enrich each frame without explicitly exchanging information across frames.Adding CFA enables cross-frame exchange, stabilizes performance, and prevents further degradation in rank metrics.
  • Temporal modeling: CTIA aggregates sentence–frame similarity over time into a video-level score and produces a clear improvement over the baseline.It organizes and propagates evidence across frames rather than relying only on a single pooled embedding.

4.6. Ablation on MoE Layers and Experts

MoE placement and expert-count ablations favor concentrating experts near the upper backbone layers and using four experts for balanced retrieval performance.

  • MoE layer placement: R@1=48.0 and R@5=74.1 make top-layer placement {11, 12} the most balanced configuration.Using only {12} gives R@1=47.9, while broader placements do not improve recall and increase MnR variance.
  • MoE layer placement: Broader MoE placement, including {7–12} and {8, 10, 12}, does not improve recall and tends to increase mean-rank variance.The ablation attributes this pattern to lower layers encoding low-level patterns and upper layers providing more disentangled semantic information.
  • Number of experts: Four experts yield the most balanced overall performance, whereas 2, 6, or 8 experts provide no consistent improvement.The visual and text encoders use identical expert counts, with NE = 4 selected as the default.

4.7. Ablation on Frame–Temporal Tokens

The FT-token ablation identifies two tokens as the strongest balanced choice, while three tokens improve R@10 without consistent gains on R@1 or R@5.

  • FT-token count: 48.0/74.1 in R@1/R@5 with MdR=2.0 makes two FT tokens the strongest balanced configuration.One token performs slightly lower, notably with higher MnR.
  • FT-token count: Three FT tokens raise R@10 to 82.1 but do not consistently improve R@1 or R@5, supporting FT=2 as the practical default.The result favors the two-token setting for overall balance rather than maximizing a single metric.

4.8. Ablation on CTIA β-Terms

The CTIA ablation evaluates complementary relevance terms, while the accompanying comparison defines the compute and profiling conventions used for model assessment.

  • CTIA term fusion: 48.0/74.1 in R@1/R@5 makes the joint setting (1, 1, 1) best for Text→Video performance.The jointly enabled terms are raw evidence, local smoothing, and graph propagation.
  • CTIA term fusion: Among single terms, rconv achieves the best Text→Video MnR at 13.5, while rgraph gives the best Video→Text result.These differing tendencies support the terms as complementary rather than interchangeable.
  • Model scale and compute: Table 8 compares model parameters, GFLOPs/sample, and retrieval performance on MSR-VTT (1k-A) and DiDeMo.GFLOPs/sample divides one profiling forward pass’s total FLOPs by batch size, with profiling frames set to 3, 8, or 12 by repository.
  • Model scale and compute: Sparse Top-K routing limits active experts per token or sample, so per-sample compute is governed primarily by K rather than scaling linearly with total experts E.The MoE design increases trainable parameters while activating only a small subset of experts.

4.10. Frame Permutation Ablation (Temporal-Order Robustness)

Frame-order perturbations cause limited performance degradation overall, with TAME more affected by complete reversal than by random shuffling.

  • Order perturbation results: TAME’s R@1 drops 1.9% under reversal, from 48.0 to 47.1, compared with CLIP4Clip’s 0.9% drop, from 44.0 to 43.6.The ablation fixes sampled frames and changes only their temporal order.
  • Order perturbation results: Under random permutation, TAME’s R@1 changes by -0.8%, from 48.0 to 47.6, comparable to CLIP4Clip’s -0.9% change.This indicates that the method’s overall effect remains largely preserved under random frame ordering.
  • Interpretation: Content-based cross-frame interaction through FT tokens and CTIA coexists with some ability to exploit directional temporal cues.This design is offered as an explanation for the slightly larger degradation under complete reversal than random shuffling.
  • Interpretation: Sparse-frame MSR-VTT queries often emphasize overall video semantics rather than strict event chronology.This protocol characteristic provides context for the limited sensitivity to temporal-order perturbations.

4.11. Masked-Frame Ablation (Robustness to Missing Temporal Evidence)

The masked-frame ablation evaluates robustness to missing temporal evidence by blacking out sampled frames at inference. TAME remains stronger than CLIP4Clip across mask ratios and degrades less under severe masking.

  • Evaluation setup: Masking randomly removes sampled frames at inference to test robustness to missing or occluded temporal evidence, not temporal-order sensitivity.The experiment uses mask ratios from 0.1 to 0.6 while keeping frame sampling and retrieval unchanged.
  • Robustness results: TAME maintains higher absolute performance than CLIP4Clip at every tested mask ratio.
  • Robustness results: 9.63% versus 11.73%: at mask ratio 0.5, TAME has a smaller relative R@1 drop than CLIP4Clip.TAME falls from 48.0 to 43.38, while CLIP4Clip falls from 44.0 to 38.84.
  • Robustness results: 16.29% versus 18.91%: at mask ratio 0.6, TAME again shows a smaller relative R@1 drop than CLIP4Clip.TAME falls from 48.0 to 40.18, while CLIP4Clip falls from 44.0 to 35.68.
  • Robustness results: At mask ratio 0.6, TAME preserves ranking quality better, with MnR increasing by 3.40 versus 5.24 for CLIP4Clip.
  • Interpretation: The final similarity combines a global video–text score with framewise evidence aggregation, supporting more stable consolidation of informative frames under partial evidence removal.

5. Conclusion

TAME extends CLIP for text–video retrieval by combining frame-wise experts, temporal tokens, and cross-temporal aggregation. It consistently outperforms CLIP-based baselines across diverse benchmarks, while requiring careful routing stabilization and selective temporal mixing to manage trade-offs.

  • Conclusion: TAME extends CLIP for text–video retrieval with frame-consistent Mixture-of-Experts, Frame–Temporal tokens, and Cross-Temporal Interaction and Aggregation.
  • Conclusion: TAME consistently outperforms CLIP-based baselines, with gains reported on MSR-VTT, DiDeMo, MSVD, LSMDC, and ActivityNet.
  • Limitations and Future Work: TAME introduces routing-stability and temporal-token-overhead trade-offs, with MoE benefits requiring stabilization and temporal mixing needing selective application to control memory cost.
  • Limitations and Future Work: CTIA relies on the quality of framewise similarities and a small set of hyperparameters despite being efficient and interpretable.
Loading 2609.02204v1…