Use What You Have: Video Retrieval Using Representations From Collaborative Experts

• Generic cues such as objects and actions are broadly available, while specific speech and on-screen text cues are less consistently available and remain challenging to exploit for retrieval.Source: Figure 1 · p. 2, §1 · pp. 2–3
• The approach hypothesizes that individual experts’ semantic representations can approximate discriminative video content, avoiding prohibitively costly large-scale text annotation for tabula rasa training.Source: §1 · p. 2
• A pairwise dynamic attention mechanism lets each expert’s relation to other experts modulate its features, such as using dog-barking audio to select dog-related RGB features.Source: §1 · p. 2
• The Collaborative Experts framework combines pretrained video embeddings into a compact video representation that is independent of the retrieval query and can be precomputed and indexed offline.Source: §1 · p. 3
• The paper evaluates representations combining available cues on multiple retrieval benchmarks and reports advances over prior work in several cases.Source: §1 · p. 3

Cross-Modal Embeddings

• Prior cross-modal work jointly embeds images and text [16] [30], and also studies audio–visual [2] [43] and audio–text embeddings; this work targets video and natural-language sentences.Video’s high-dimensional content makes the video–language embedding problem more challenging.Source: §2 · p. 3

Video-Text Embeddings

• Video–text methods often focus on single frames; mean-pooling an image–text model’s video-frame features outperformed many prior approaches, but misses motion, audio, and speech cues [12] [16].Other methods use audio and action-recognition motion features, but do not exploit speech-to-text or OCR; those cues have been used in lecture and news retrieval [39].Source: §2 · p. 3
• The approach extends the joint embedding of [39], built on Mixtures-of-Experts, to investigate speech and text cues and use pretrained features through robust collaborative gating.The framework temporally aggregates pretrained video experts and gates their collaboration; text-query and video embeddings are computed independently to maintain retrieval efficiency.Source: §2 · p. 3, Figure 2 · p. 4

Annotation scarcity

• Video retrieval is constrained by small training datasets because natural-language video annotation is costly, motivating reuse of pretrained embeddings from other tasks.Semantic visual experts have been used for image retrieval and video retrieval [39]; alternatives include self-supervised [54] and weakly-supervised [70] video–text models.Source: §2 · p. 3

3.1 Video Encoder

• The video encoder pools frozen pretrained experts, projects them to a shared dimensionality, and uses collaborative gating to select complementary information and suppress noise or conflicts.Average pooling aggregates slower visual features, while NetVLAD [3] aggregates more dynamic audio and word features. Shared MLPs model pairwise relationships among expert projections to compute attention; gated responses pass through GEMs and are concatenated into a fixed-length video embedding.Source: §3.1 · p. 5

3.2 Text Query Encoder and Training Loss

• Text queries aggregate frozen contextual word embeddings, then GEM projections are scaled by softmax mixture weights and concatenated to match the video embedding dimensionality.The mixture weights are one scalar per expert projection and are computed from the aggregated text features.Source: §3.2 · pp. 5–6
• The video and text encoders are learned jointly using a bidirectional max-margin ranking loss, and retrieval similarity is computed as the inner product of their embeddings.The embeddings’ inner product is also the cosine similarity because the encoded outputs are L2-normalized.Source: §3.2 · p. 6

3.2.1 Missing Experts

• When experts are missing, their features are zero-padded and their mixture weights are removed before the remaining weights are renormalized to sum to one.This compensates for the implicit scaling caused by computing similarity between shorter embeddings.Source: §3.2.1 · p. 6

4.1 Datasets, Implementation Details and Metrics

• Experiments use MSR-VTT, LSMDC, MSVD, DiDeMo, and ActivityNet-captions, reporting standard rank and R@K metrics and means and standard deviations across three seeded runs.For video-to-sentence evaluation on MSR-VTT and MSVD, the protocol uses the minimum rank among valid descriptions for each video query.Source: §4.1 · pp. 6–7
• CE combines pretrained appearance, motion, scene, face, audio, speech, and OCR features; visual features are averaged over time, while speech, audio, and OCR use NetVLAD aggregation.Features are projected to 768 dimensions; text uses word2vec embeddings contextualized by pretrained OpenAI-GPT and aggregated with NetVLAD.Source: §4.1 · pp. 6–7
• MSVD uses only features available without audio; LSMDC reuses existing face, text, and audio features and combines them with the other described features.The CE framework is implemented in PyTorch and optimized with Lookahead and RAdam.Source: §4.1 · p. 7

4.2 Comparison to Prior State-of-the-Art

• CE improves both MSR-VTT retrieval directions, matches or exceeds prior work on LSMDC and MSVD, outperforms prior work on DiDeMo, and is competitive on ActivityNet.The ActivityNet comparison is limited to methods using the same video-level annotation setting.Source: §4.2 · pp. 7–8, Table 1 · p. 8, Table 2 · p. 8

4.3 Ablation Studies

• Stronger same-modality features can significantly boost performance: Instagram [37] rather than Kinetics [6] for action experts, and ImageNet for object experts; scarce speech, face, and OCR cues add limited improvement.The expert comparison combines these features with a consistently available scene-expert baseline because some experts are absent from videos.Source: §4.3 · p. 9, Table 5 · p. 9
• Collaborative gating improves performance and enables more efficient parameterization than the compared aggregation alternatives, including MoEE and CE without collaborative gating.The study also compares simple expert concatenation and CE without common-dimension projection, mixture weights, or collaborative gating.Source: §4.3 · p. 9, Table 6 · p. 10
• Increasing training captions from 1 to 20 improves performance approximately as much as adding the full expert collection, and using both captions and experts yields the most robust embedding.The qualitative examples also show that rankings can confuse details such as car color, while removing CE produces noisier rankings.Source: §4.3 · pp. 9–10, Table 6 · p. 10, Figure 3 · p. 10

5 Conclusion

• Collaborative experts learns a joint video-text embedding for efficient retrieval, with pretrained features combined through gating to boost retrieval performance.Source: §5 · p. 10
• Future work will explore collaborative experts for video understanding tasks such as clustering and summarisation.Source: §5 · p. 10

A.1 Paper update, result corrections and summary of differences

• The initial implementation bug overestimated model performance and led to inaccurate conclusions about the relative importance of different experts.The corrected paper repeats the experiments after removing the bug and updates its written analysis to reflect the changed results.Source: §A.1 · p. 15
• The correction adds systematic hyperparameter selection and pretrained experts to assess feature strength within a modality.It repeats the initial experiments with the bug removed and updates the written analysis to reflect the results.Source: §A.1 · p. 15
• During training and testing, the bug leaked target-video feature availability to the query encoder through incorrect embedding-distance weighting.The causes were a leaking broadcasting operation in an imported library [39] and incorrect NaN handling; the bug was patched in known codebases using the implementation.Source: §A.1 · p. 15

MSR-VTT

• MSR-VTT has about 200K caption pairs for 10K diverse YouTube clips, with train/validation/test splits of 6,513/497/2,990 videos.The authors note duplicate-caption label noise and also evaluate prior-work splits; for [39]’s split without validation, they train for 100 epochs before measuring performance.Source: §A.2 · pp. 15–16

MSVD

• MSVD has 80K English descriptions for 1,970 YouTube videos, uses the 1,200/100/670 standard split, and has no audio streams.Each video has around 40 captions.Source: §A.2 · p. 16

LSMDC

• LSMDC contains 118,081 clips from 202 movies, each captioned from a script or transcribed descriptive video service; evaluation uses 7,408 validation and 1,000 test clips.The test movies are disjoint from the training and validation sets, following the LSMDC challenge protocol.Source: §A.2 · p. 16

ActivityNet-Captions

• ActivityNet Captions [32] comprises 20K YouTube videos and about 100K descriptive sentences; paragraph-video retrieval follows, training up to 200 epochs and reporting val1 results.The stated train/test split allocates 10,009 and 4,917 videos, respectively.Source: §A.2 · p. 16

DiDeMo

• DiDeMo [1] contains 10,464 unedited personal videos with 40,543 descriptions and timestamped moments, but this work does not use the timestamps.Videos have roughly 3–5 description/moment pairs each and diverse open-world content.Source: §A.2 · p. 16

A.3 Optimisation details and hyperparameter selection

• Per-dataset single-expert grid searches were followed by 1k-configuration MSR-VTT Asynchronous Hyperband [33] tuning of expert and gating settings.The initial search used Lookahead over batch size, learning rate, and weight decay; projection tests favored 768 over 512 and 1024 dimensions, fixed for later experiments despite low sensitivity. MoEE [39] also received a 1k-configuration budget for a fair MSR-VTT comparison.Source: §A.3 · p. 16
• DiDeMo, MSVD, and LSMDC received further 500-configuration searches, while ActivityNet used the DiDeMo settings because its validation set assesses performance.Configurations, experts, pretrained models, and experiment logs are available in the updated open-source implementation.Source: §A.3 · pp. 16–17

A.4 Ablation Studies - Full Tables

• Table 7 examines the importance of each expert when combined with Scene features.Source: Table 7 · p. 17
• Table 8 examines the importance of experts on MSR-VTT.Source: Table 8 · p. 17
• Table 9 examines how the number of training captions affects MSR-VTT ablations.Source: Table 9 · p. 17

A.5 Implementation Details

• Object-frame embeddings use Obj(IN) and Obj(IG), while action and face embeddings come from separate pretrained visual models.Obj(IN) uses SENet-154 pretrained on ImageNet, and Obj(IG) uses ResNext-101 pretrained on Instagram hashtags [37]; both process frames at 25 fps resized to 224 × 224 and yield 2048-dimensional features. Action models are I3D and R(2+1)D, while face embeddings are 512-dimensional ResNet50 [22] features extracted after SSD detection [35] and trained on VGGFace2 [5].Source: §A.5 · pp. 17–18
• Audio embeddings use VGGish, and 2208-dimensional scene embeddings use DenseNet-161 [25] pretrained on Places365.Audio is resampled to 16 kHz mono, converted to a log-mel spectrogram, and parsed into non-overlapping 0.96-second segments mapped to 128-dimensional vectors; scene frames are sampled at 1 fps.Source: §A.5 · p. 18
• Speech transcripts are generated for MSR-VTT, MSVD, and ActivityNet, while detected on-screen text is recognized and encoded with pretrained word2vec [41].The English Google Cloud Speech to Text API processes resampled audio; MSR-VTT has audio for 8,811 of 10,000 videos, with speech detected in 5,626. OCR uses Pixel Link [10] detection and a Synth90K-trained recognizer.Source: §A.5 · p. 18
• Sentence text is encoded with Google News word2vec embeddings [41], contextualized by pretrained OpenAI-GPT, and aggregated using NetVLAD.Source: §A.5 · p. 19