PaperlayerView exact revision on arXiv

Source-linked AI summary

Vote-in-Context: Turning VLMs into Zero-Shot Rank Fusers

Mohamed Eltahir, Ali Habibullah, Lama Ayash, Tanveer Hussain, Naeemullah Khan

arXiv:2511.01617v1cs.CVcs.IR

TL;DR

Multimodal retrieval is challenging, and conventional fusion uses rank or score signals without considering candidate content. ViC prompts a frozen VLM with content and retriever metadata, using S-Grids for video candidates. Across benchmarks, it establishes state-of-the-art zero-shot retrieval performance.

  • Problem

    Multimodal retrieval remains challenging, while conventional fusion methods operate on rank or score signals and ignore candidate content.

    Source: §1 · p. 2

  • Method

    ViC serializes content evidence and retriever metadata in a frozen VLM's prompt, using S-Grids to represent video candidates for list-wise reranking and fusion.

    Source: §1 · p. 2

  • Results

    Across benchmarks, ViC establishes state-of-the-art zero-shot retrieval, improves single-backbone reranking, and outperforms RRF and CombSUM in fusion.

    Source: §1 · p. 2, §5.4 · p. 10

  • Takeaways & Limitations

    S-Grid processing scales with candidate count K rather than raw video length, supporting retrieval-quality improvements under fixed latency constraints.

    Source: §5.4 · p. 10

    Current VLMs' effective context windows limit list-wise relevance judgments; for v2t, performance plateaus or declines beyond 20 captions.

    Source: §5.3.3 · p. 10

Abstract

from arXiv · show

In the retrieval domain, candidates' fusion from heterogeneous retrievers is a long-standing challenge, particularly for complex, multi-modal data such as videos. While typical fusion techniques are training-free, they rely solely on rank or score signals, disregarding candidates' representations. This work introduces Vote-in-Context (ViC), a generalized, training-free framework that re-thinks list-wise reranking and fusion as a zero-shot reasoning task for a Vision-Language Model (VLM). The core insight is to serialize both content evidence and retriever metadata directly within the VLM's prompt, allowing the model to adaptively weigh retriever consensus against visual-linguistic content. We demonstrate the generality of this framework by applying it to the challenging domain of cross-modal video retrieval. To this end, we introduce the S-Grid, a compact serialization map that represents each video as an image grid, optionally paired with subtitles to enable list-wise reasoning over video candidates. ViC is evaluated both as a single-list reranker, where it dramatically improves the precision of individual retrievers, and as an ensemble fuser, where it consistently outperforms strong baselines like CombSUM. Across video retrieval benchmarks including ActivityNet and VATEX, the framework establishes new state-of-the-art zero-shot retrieval performance, demonstrating its effectiveness in handling complex visual and temporal signals alongside text. In zero-shot settings, ViC achieves Recall@1 scores of 87.1% (t2v) / 89.0% (v2t) on MSR-VTT and 99.6% (v2t) on VATEX, representing massive gains of up to +40 Recall@1 over previous state-of-the-art baselines. We present ViC as a simple, reproducible, and highly effective recipe for turning modern VLMs into powerful zero-shot rerankers and fusers. Code and resources are publicly available at: https://github.com/mohammad2012191/ViC

1. Introduction

Complex multimodal data makes retrieval difficult, while conventional fusion ignores candidate content; ViC prompts a frozen VLM with content evidence and retriever metadata. For video retrieval, it uses S-Grids to provide VLM-readable candidate evidence.

1. Introduction

  • Two-stage retrieval uses efficient retrievers to shortlist candidates before costly reranking [4], and multi-retriever outputs are often fused with RRF or CombSUM/CombMNZ.This fusion typically offers significant performance gains.Source: §1 · p. 2
  • First-stage retrievers can miss query-specific details, conventional rerankers may be costly or require in-domain fine-tuning, and rank/score fusion ignores candidate content.These limitations motivate a training-free framework that can rerank and fuse with content awareness.Source: §1 · p. 2
  • ViC uses a frozen VLM as a training-free list-wise reranker and fuser, serializing content and retriever metadata in its prompt for adaptive weighting.The metadata includes per-list ranks and cross-list multiplicity.Source: §1 · p. 2
  • S-Grid represents each video as a single image grid of uniformly sampled frames, optionally paired with subtitles, to provide VLM-readable content evidence.The grid is the content serialization map used for video retrieval.Source: §1 · p. 2
  • ViC improves all single backbones as a reranker and consistently outperforms RRF and CombSUM as a fuser.The paper evaluates both single-list (M = 1) and fusion (M > 1) modes.Source: §1 · p. 2
  • The paper releases its framework and evaluation protocols, including analyses of scaling, context-size sensitivity, and candidate assembly strategies.The contributions also include comprehensive evaluation in single-list reranking and fusion modes.Source: §1 · p. 2

2. Related Work

Prior video retrievers use coarse global representations for efficient candidate selection, but may miss fine-grained query details. Existing fusion methods aggregate rank or score signals, while list-wise language-model reranking has prompted efforts to adapt VLMs to video.

2. Related Work

  • First-stage video retrievers efficiently narrow the search space with coarse global similarity, but can struggle to capture fine-grained, query-specific details.The cited approaches span CLIP-style adaptations and broader multimodal or video-specific systems.Source: §2 · p. 3
  • Classical fusion methods aggregate scores with CombSUM/CombMNZ or ranks with RRF, but rely on fixed formulas and use only rank or score signals.RRF may use a fixed hyperparameter such as k.Source: §2 · p. 3
  • LLMs have shown zero-shot list-wise reranking in text retrieval [5] [22], while sampled-frame grids let image-centric VLMs reason about video temporal dynamics [23].Instruction-following VLMs such as InternVL [11] and Qwen-VL [12] provide multimodal alignment capabilities.Source: §2 · p. 3

3. Methodology

ViC uses a frozen VLM to rerank or fuse candidate lists by presenting both candidate content and retriever metadata in a prompt. For video retrieval, S-Grid serializes videos as a compact visual-linguistic representation for VLM reasoning.

3. Methodology

  • ViC gives a frozen VLM both candidate content and retriever metadata, letting it adaptively weigh these signals for list-wise reranking and fusion.Rank is conveyed by list order and consensus by candidate multiplicity.Source: §3 · pp. 3–4

3.1. Problem Setup and Notation

  • Multiple ranked lists are truncated and round-robin interleaved into a candidate sequence whose positions and duplicates preserve rank and cross-list consensus.The VLM returns a permutation that reorders the sequence into the fused output.Source: §3.1 · pp. 4–5
  • With one retriever, ViC reranks its top-K list using candidate content alone, without cross-list multiplicity or rank-of-ranks signals.The VLM call and final output remain the same as in fusion mode.Source: §3.1 · p. 5

3.2. Applying ViC to Video Retrieval.

  • S-Grid tiles uniformly sampled video frames into one image and can include subtitles as auxiliary text for the VLM.The design provides visual snapshots and audio transcripts within a single prompt.Source: §3.2 · p. 5
  • Video retrieval applies the framework in both text-to-video and video-to-text directions, using video serialization for video inputs and identity mapping for text candidates.Processing cost scales as O(K·CVLM) and is independent of raw video length because each video is represented by one image.Source: §3.2 · p. 5

3.3. List Fusion Strategies

  • Unlike score fusion or RRF, ViC lets the VLM weigh content, rank, and multiplicity per query rather than collapsing lists into a fixed aggregate score.Soft Voting uses calibrated similarity scores, while RRF is presented for heterogeneous ranked lists.Source: §3.3 · p. 6

4. Experiments

The experiments evaluate ViC on four video-retrieval benchmarks with frozen first-stage retrievers and a zero-shot VLM reranker. They report R@1 for single-list reranking and ensemble fusion under specified candidate and evaluation protocols.

4.1. Benchmarks and Protocol

  • Evaluation covers MSR-VTT, DiDeMo, ActivityNet Captions, and VATEX, using the stated standard retrieval protocols and benchmark-specific splits or subsets.VATEX uses a fixed subset of 1,252 recovered videos, with baselines and ViC reproduced on the same subset.Source: §4.1 · p. 6

4.2. Implementation Details

  • CLIP4Clip, VAST, GRAM, and InternVideo2-6B serve as frozen first-stage retrievers, with official checkpoints or configurations and repository-consistent preprocessing.InternVL 3.5 38B is the main VLM reranker, used zero-shot without dataset-specific fine-tuning; the standard ensemble excludes VAST, while VATEX uses InternVideo2 and VAST.Source: §4.2 · p. 6

4.3. Metrics and Hyperparameter

  • Ensemble candidate lists interleave retriever rankings up to depth kmax while preserving duplicates, but evaluation counts only each candidate’s highest-ranked instance.Soft Voting min-max normalizes similarity scores per query before uniform-weight aggregation.Source: §4.3 · pp. 6–7
  • R@1 is reported in both t2v and v2t directions; Table 1 compares single backbones with and without ViC reranking, while Table 2 covers ensemble fusion methods.The stated candidate counts are K = 14 S-Grids for t2v and K = 20 captions for v2t; the default S-Grid contains 3 × 3 frames.Source: §4.3 · p. 6, Table 1 · p. 7, Table 2 · p. 7

5. Results

In zero-shot video retrieval, ViC reranks single-retriever lists and fuses multiple lists using video content and retriever metadata. It improves R@1 over individual backbones and traditional fusion baselines, while experiments also examine grid size, model scale, and context limits.

5.1. ViC as a Single-List Reranker (M = 1)

  • ViC raises MSR-VTT (t2v) CLIP4Clip from 34.4 to 64.2 (+29.8 points), InternVideo2 from 54.5 to 75.9 (+21.4), and ActivityNet InternVideo2 from 58.2 to 89.8 (+31.6).On VATEX (v2t), ViC boosts VAST from 77.6 to 99.6 (+22.0 points).Source: §5.1 · p. 7

5.2. ViC as an Ensemble Fuser (M > 1)

  • On MSR-VTT (t2v), ViC achieves 87.1 R@1, +1.8 points over CombMNZ; on DiDeMo (t2v), 87.4 R@1 is +7.0 points over CombSUM, and VATEX (t2v) reaches 97.5 R@1.On MSR-VTT, ViC fusion reaches 87.1 versus 75.9 for the best single-backbone reranking result, an additional +11.2 points.Source: §5.2 · pp. 8–9
  • Removing duplicate candidates drops MSR-VTT (t2v) R@1 from 87.1 to 84.2, consistent with ViC using cross-list consensus as a relevance signal.The no-duplicates variant removes candidate multiplicity metadata before VLM processing.Source: §5.2 · p. 9

5.3.1. Grid size

  • Grid sizes 2 × 2 and 3 × 3 are sweet spots; 1 × 1 undercovers video, while 4 × 4 compresses frames aggressively and can add redundant visual tokens.The trend holds across tested benchmarks; reranking also performs strongly on ActivityNet’s longer, untrimmed videos.Source: Figure 6 · p. 8, §5.3.1 · p. 9

5.3.2. Reranker scale

  • At a fixed 3×3 grid, scaling InternVL 3.5 from 8B to 38B steadily improves t2v R@1 before saturation; smaller models fail to produce consistent permutations.The authors identify 8B as the minimum effective scale for zero-shot list-wise reranking.Source: Figure 6 · p. 8, §5.3.2 · p. 9

5.3.3. VLM Type and Context size

  • Larger lists show diminishing returns: t2v K=10→14 raises R@1, but K=30 lowers R@1 for negligible R@10 gain.For v2t, K=20 captions is the most effective operating point; Qwen3-VL deteriorates on v2t as context grows, while Gemma-3 remains stable at K=30. InternVL 3.5 is used in the main experiments for consistent overall performance and multiple scale variants.Source: §5.3.3 · pp. 9–10

5.4. Discussion and Conclusion

  • ViC adaptively balances rank and multiplicity metadata against content evidence per query, while S-Grid processing cost scales with candidate count K rather than raw video length.The video application uses compact grids of uniformly sampled frames, optionally paired with subtitles.Source: §5.4 · p. 10
  • ViC trades higher inference cost than RRF or CombSUM for improved performance, and video reranking cannot recover relevant items absent from the first-stage top-K list.Uniform frame sampling may also miss short, semantically important events in long, untrimmed videos.Source: §5.4 · p. 10
Loading 2511.01617v1…