PaperlayerView exact revision on arXiv

Source-linked AI summary

VEDJE: Video-Efficient Discriminative Joint Encoder for Scalable Video-Text Retrieval

Shahaf Wagner, Gabriele Serussi, Dan Ben Ami, Tomer Galanti, Chaim Baskin

arXiv:2610.11850v1cs.CV

TL;DR

Video reranking must distinguish different events in similar scenes, but rich video representations make query-time verification costly. VEDJE compresses features within sampled frames into a reusable cache and adds training-only feature-change supervision. Across four benchmarks it improves R@1 over matched first-stage retrievers in both directions.

  • Problem

    Video search must verify that candidates match queried actions and moments despite similar scenes, while rich representations and joint scoring make query-time verification costly.

    Source: §1 · pp. 1–2

  • Method

    VEDJE stores separate compressed representations for sampled frames and uses feature-change supervision to improve retrieval without adding query-time computation.

    Source: §1 · p. 2, §3 · p. 3, §3.3 · p. 4

  • Results

    Across MSR-VTT, MSVD, DiDeMo, and ActivityNet, VEDJE improves R@1 over matched first-stage retrievers in both retrieval directions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    VEDJE supports joint video search using compact evidence computed once and reused for later queries.

    Source: §5 · p. 9

    The evaluations use English captions and web-sourced videos and do not establish performance on multilingual video retrieval.

    Source: §5 · p. 9

Abstract

from arXiv · show

Finding the right video often requires distinguishing similar scenes in which different events occur. Joint matching improves retrieval, but processing rich video representations for each query is costly. VEDJE compresses features within sampled frames while keeping their representations separate in a reusable cache. Feature-change prediction supplies an auxiliary training signal that improves retrieval from the compressed cache without adding work at query time. On MSR-VTT, MSVD, DiDeMo, and ActivityNet, VEDJE improves R@1 over matched first-stage retrievers in both retrieval directions. On MSR-VTT, it reaches 59.8 text-to-video R@1 with a fine-tuned VideoCLIP-XL first stage. In the VideoPrism configuration, shrinking the per-video cache fourfold to 12 KiB preserves text-to-video recall within 0.2 points. These results show that accurate video search can operate on compact evidence, encoded once and reused as new queries arrive.

1 INTRODUCTION

Video reranking must distinguish actions and moments in visually similar scenes, but rich video features and joint scoring make query-time verification costly. VEDJE uses a compact, reusable frame-indexed cache and feature-change supervision; it improves R@1 over matched first-stage retrievers across four benchmarks.

1 INTRODUCTION

  • Video verification must distinguish videos with similar scenes but different actions, state changes, or queried moments.The second stage checks whether retrieved candidates match the query.Source: §1 · p. 1
  • Video rerankers face growing stored features and joint-scoring sequences as sampled frames increase, while online candidate re-encoding incurs severe latency.Multimodal LLM rerankers put multi-billion-parameter decoders directly on the query path.Source: §1 · p. 2
  • VEDJE compresses features within sampled frames while retaining separate representations across sampled times for a lightweight joint encoder to read with the query.Feature-change supervision improves retrieval from compressed tokens without adding query-time computation.Source: §1 · p. 2
  • A 33M-parameter cross-encoder scores cached tokens with query text and the first-stage score, while the token budget sets both cache storage and online visual sequence length.A frozen visual encoder processes each video once offline.Source: §1 · p. 2
  • Across MSR-VTT, MSVD, DiDeMo, and ActivityNet, VEDJE improves R@1 over matched first-stage retrievers in both retrieval directions.With a fine-tuned VideoCLIP-XL first stage, it reaches 59.8 text-to-video R@1 on MSR-VTT; the VideoPrism configuration uses a fourfold-smaller 12 KiB cache while preserving T2V R@1 within 0.2 points.Source: §1 · p. 2

2 RELATED WORK

Prior video-text retrieval work trades cheap independent matching against richer query-time interaction, while some joint rerankers require large query-path models. VEDJE positions cached joint reranking as a compact video extension of the image-caching recipe.

Dual encoders for video-text retrieval

  • Dual encoders retrieve candidates efficiently through inner-product similarity, but fix video representations before queries arrive and provide no cross-modal interaction.CLIP-style retrieval stages include Luo et al. (2022), Wang et al. (2024a;b), and Zhao et al. (2024).Source: §2 · p. 2

Late interaction and multi-vector matching

  • Late interaction retains richer unimodal token banks but does not let query and cached tokens share a joint encoder; EERCF instead uses temporal encoding and text-gated aggregation.VEDJE is compared with EERCF on shared CLIP ViT-B/16, and is separately evaluated on candidates from its released ViT-B/32 retriever.Source: §2 · p. 2

Joint rerankers for vision-language retrieval

  • VEDJE adapts EDJE’s image-caching recipe to video by retaining separate sampled-time representations and applying lightweight joint attention with a 33M encoder.CrossTVR uses multi-grained cross-attention; MLLM rerankers put multi-billion-parameter decoders on the query path.Source: §2 · pp. 2–3

Where VEDJE sits

  • VEDJE’s matched-budget comparisons test separate frame groups against pooling, while cache-size ablations measure feature-change supervision’s retrieval effects.Its closest neighbour is EDJE’s image-caching recipe.Source: §2 · p. 3

3 METHOD

VEDJE replaces query-time video encoding with a compact cache written offline and read by a lightweight joint reranker. Its method combines frame-indexed compression, training-only feature-change supervision, and a residual first-stage score.

3 METHOD

  • VEDJE writes a compact video cache offline, trains with a feature-change head that is discarded, and reranks online with a 33M joint encoder.The reranker combines query and cache representations with the embedded first-stage score before its nonlinear score head.Source: Figure 2 · p. 3, §3 · p. 3

3.1 SETUP AND NOTATION

  • A first-stage retriever supplies candidate videos and coarse scores; each candidate is sampled into T frames and encoded into frame-indexed patch features for a cache of M tokens per frame.The product TM controls per-video storage and the online video sequence length.Source: §3.1 · pp. 3–4

3.2 A FRAME-INDEXED CACHE ACROSS THE SAMPLED SEQUENCE

  • A shared decoder compresses each frame’s projected patch features into M tokens, then concatenates frame groups in temporal order.This bounds cache storage by T × M tokens independently of patch count P while retaining a separate group for each sampled time.Source: §3.2 · p. 4

3.3 FEATURE-CHANGE SUPERVISION FOR A TIGHT CACHE

  • Feature-change supervision trains a predictor to estimate differences between frozen visual features at sampled times from the compressed frame cache.The loss averages squared error over valid time-horizon pairs; the training-only predictor is discarded, adding no retrieval-time computation.Source: §3.3 · p. 4

3.4 CACHED JOINT RERANKING WITH A RESIDUAL PRIOR

  • The reranker jointly encodes query text and cached video tokens, then adds an embedding of the first-stage score to their pairwise representation before nonlinear scoring.Its learned positional embedding spans the full text-and-cache input.Source: §3.4 · p. 5

3.5 TRAINING AND DEPLOYMENT

  • Training jointly optimizes the compressor, joint encoder, prior embedding, score head, and auxiliary heads with four objectives.After training, auxiliary heads are discarded; changing the visual backbone or compressor requires rebuilding the cache.Source: §3.5 · p. 5

4 EXPERIMENTS

The experiments evaluate retrieval quality, cache design, and complete-query cost through matched comparisons of frame grouping, feature-change supervision, and joint scoring. They cover four datasets and assess text-to-video and video-to-text retrieval.

4 EXPERIMENTS

  • Matched controls test how frame grouping, feature-change supervision, and joint scoring contribute to retrieval quality, cache design, and full-query cost.The experimental comparisons include complete-query latency measurements alongside retrieval results.Source: §4 · p. 5, §4.7 · p. 8

4.1 EXPERIMENTAL SETUP

  • VEDJE is evaluated on four standard-split benchmarks using Recall@K in both retrieval directions, with a default top-20 candidate pool and a 64-token BF16 cache.The default setup uses VideoPrism features, a 33M-parameter joint reranker, and prediction horizon H = {3}.Source: §4.1 · pp. 5–6

4.2 Q1: DOES RERANKING IMPROVE RETRIEVAL ACROSS SETTINGS?

  • 56.2 to 59.8 T2V R@1 and 55.1 to 58.8 V2T R@1 with fine-tuned VideoCLIP-XL; VEDJE also improves R@1 in both directions across four datasets.The cross-dataset evaluation covers MSR-VTT, MSVD, DiDeMo, and ActivityNet; the fine-tuned VideoCLIP-XL gains remain after first-stage fine-tuning.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 Q2: HOW SMALL CAN THE CACHE BECOME?

  • 0.2 points lower T2V R@1 (54.6 to 54.4) accompanies a fourfold cache reduction from 48 KiB to 12 KiB per video.V2T R@1 falls from 54.6 to 53.1 in the same comparison, so the compact cache carries a larger accuracy cost for V2T.Source: §4.3 · p. 7, Table 3 · p. 7

4.4 Q3: WHICH CACHE STRUCTURE CARRIES THE GAIN?

  • 2.0 points higher R@1 in both directions over attention pooling comes from retaining separate frame groups at the same storage budget.Feature-change supervision raises recall further to 54.6 in both directions.Source: §4.4 · p. 7, Table 4 · p. 8

4.5 Q4: WHERE DOES FEATURE-CHANGE SUPERVISION HELP?

  • 1.9-point T2V R@1 gain with 16 tokens, versus 0.6 points with 64 tokens, shows larger measured T2V gains from feature-change supervision at the tighter budget.V2T shows the opposite budget pattern, with a larger gain at 64 tokens; at 64 tokens, the delta target reaches 54.6 in both directions versus 54.2 T2V and 53.9 V2T for current-frame reconstruction.Source: Table 3 · p. 7, §4.5 · p. 8

4.6 Q5: WHAT DO JOINT SCORING AND THE PRIOR CONTRIBUTE?

  • 54.6 T2V R@1 for the joint encoder exceeds trained MaxSim's 51.9 when both score the same cache and candidate pool.The full model reaches 54.6, compared with 50.1 for prior-only and 43.8 for joint-only models.Source: §4.6 · p. 8, Table 5 · p. 8

4.7 Q6: WHAT DOES A COMPLETE QUERY COST?

  • 6.33 ms at 100 candidates with the 16-token cache, versus 7.83 ms with 64 tokens, shows the latency difference grows as more candidates are scored.The comparison measures complete-query cost; reducing the cache also reduces the video payload read per query.Source: §4.7 · p. 9, Table 6 · p. 9

5 CONCLUSION

VEDJE’s conclusion centers on compact, frame-separated evidence reusable across future queries. The appendix documents its evaluation protocol and the offline-indexing and online-reranking sequence.

5 CONCLUSION

  • Across four benchmarks, the reusable cache supports gains over first-stage retrieval, including with a fine-tuned first stage.The conclusion presents compact evidence computed once and reused for later queries as a practical route to joint video search.Source: §5 · p. 9
  • The cache is query-blind, backbone and compressor updates require re-indexing, and the first and second stages are not jointly trained.The evaluations use English captions and web-sourced videos, so they do not establish multilingual retrieval performance.Source: §5 · p. 9

A ADDITIONAL EXPERIMENTAL DETAILS

  • Appendix controls cover deployment details and secondary experiments, while recall values and gains are summarized using the stated statistical protocol.The protocol describes means, population standard deviations, and a separate four-run check for the default-budget feature-change loss.Source: A ADDITIONAL EXPERIMENTAL DETAILS · p. 14

A.1 VEDJE INDEXING AND RERANKING ALGORITHM

  • Indexing stores ordered compressed tokens for each sampled frame; reranking fetches candidates’ tokens and scores query-video pairs without running the visual backbone online.Text-to-video retrieval sorts candidates by the joint score, and the algorithm specifies that the visual backbone is not on the query path.Source: §A.1 · p. 14

A.2 ARCHITECTURE DETAILS

The architecture compresses frozen video features into frame-indexed tokens for an online joint reranker. Training jointly optimizes four objectives, while quantization controls and compute and asset notes document deployment and evaluation conditions.

A.2 ARCHITECTURE DETAILS

  • The frozen backbone supplies frame-indexed features; a shared decoder compresses each frame to four tokens by default for a MiniLM-L12-H384 joint reranker.With 16 frames, the default cache contains 64 tokens; the reranker concatenates cached video embeddings with text input embeddings.Source: §A.2 · p. 15

A.3 TRAINING PROTOCOL

  • Training jointly optimizes matching, contrastive, masked-language, and future-delta objectives; auxiliary heads are discarded at deployment.The future-delta objective regresses patch-level changes against frozen-backbone features, and the visual backbone remains frozen during training.Source: §A.3 · pp. 15–16

A.4 CACHE QUANTIZATION

  • Cache quantization controls store FP8-E4M3 or FP4-E2M1 values and dequantize them to FP32 before joint encoding; reranker weights retain training precision.Storage counts the packed cache payload, excluding metadata.Source: §A.4 · p. 16

A.5 COMPUTE AND CODE RELEASE

  • Training and indexing use one 48 GB NVIDIA L40S; the estimated 850 to 900 GPU-hours exclude preliminary and failed runs and do not fully account for additional controlled comparisons.The estimate covers initial configurations, seed repeats, and appendix ablations.Source: §A.5 · p. 16

A.6 END-TO-END LATENCY AND CACHE I/O

The latency and storage measurements quantify reranking cost, cache-transfer share, and payload reductions under specified hardware and cache conditions.

A.6 END-TO-END LATENCY AND CACHE I/O

  • 3.03/1.73 ms is the reranker latency for 64/16 tokens at Nc=100, with 0.99/0.78 GB peak resident memory.Measurements use one NVIDIA L40S; end-to-end timing includes prior processing, cache fetching and transfer, and joint scoring.Source: §A.6 · pp. 16–17, Table 8 · p. 17
  • 2.4%/1.2% of end-to-end query time at Nc=20 goes to host-cache fetching and copying for 64/16 tokens; the share stays at most 5.5% through 100 candidates.Warm NVMe reads approach host-RAM latency, while cold reads are slower.Source: §A.6 · p. 17
  • 12.3/49.2 GB per million videos is the extrapolated payload for 16/64 BF16 tokens, excluding container metadata and the first-stage index.These are storage extrapolations, not measured end-to-end corpus sizes.Source: §A.6 · p. 17
  • 48 KiB for 64 tokens replaces 6.02 MiB of full visual features per video, a 128.5-fold payload reduction; the 16-token cache uses 12 KiB.These payload figures exclude the first-stage index and file metadata.Source: §A.6 · p. 17, Table 9 · p. 17

A.7 CONTROLLED COMPARISON PROTOCOLS

The controlled comparisons examine a single-frame reference, feature-change training, same-cache scoring, and reranking candidates from a released EERCF retriever.

A.7 CONTROLLED COMPARISON PROTOCOLS

  • 29.6 ± 0.4 T2V and 31.4 ± 0.6 V2T R@1 are the single-frame EDJE-style reference results on the same VideoPrism candidates.These are means and standard deviations over three training seeds, reported separately from the matched multiframe cache comparison.Source: §A.7 · p. 17
  • +0.60/+1.00 R@1 points are the mean T2V/V2T gains with Ldelta across four MSR-VTT runs at the default 64-token budget.The standard deviations of paired gains are 0.13/0.11, respectively.Source: §A.7 · p. 17
  • The same-cache scoring comparison holds VideoPrism features, a 64-token cache, first stage, and 20-candidate pool constant.The trained MaxSim head follows VEDJE’s training data and schedule; raw VideoPrism supplies the stage-1 reference.Source: §A.7 · p. 18
  • 42.8 to 49.1 T2V R@1: VEDJE reranks the released EERCF CLIP ViT-B/32 candidate pool; EERCF’s published reference is 47.8 under its own protocol.The comparison uses 50 candidates per query on MSR-VTT.Source: §A.7 · p. 18, Table 10 · p. 18

Method

VEDJE evaluates reranking with both a released candidate generator and a fine-tuned VideoCLIP-XL first stage. The reported results include matched first-stage controls and a comparison with EERCF’s published result.

Method

  • On released-checkpoint candidates, VEDJE scored 49.1, compared with 42.8 for the reproduced released coarse checkpoint and 47.8 for EERCF’s published result.Source: Method · p. 18
  • 59.8/58.8 reranked T2V/V2T R@1 with VideoCLIP-XL, versus 56.2/55.1 before reranking.The first-stage T2V R@1 was 56.2, and the reranked T2V/V2T R@1 was 59.8/58.8.Source: Method · p. 18
  • Table 11 separates prior-only and joint-only models trained independently, while the full model combines the prior with joint text-video features.The table caption describes stage-1 prior controls on MSR-VTT 1ka with fixed candidates.Source: Table 11 · p. 18

A.8 FULL RETRIEVAL RESULTS

The appendix reports retrieval results across datasets and directions, then examines candidate-pool size, cache precision, future-delta supervision, and prediction targets. These controls show how retrieval varies with candidate count and training or cache choices.

A.8 FULL RETRIEVAL RESULTS

  • Tables 13–15 report Recall@K separately for text-to-video and video-to-text across four datasets, with Table 15 giving matched first-stage comparisons and MSR-VTT three-seed T2V R@1 of 54.6 ± 0.1 for VP and 56.5 ± 0.2 for VCLIP.The variant conventions identify VP, VCLIP, and PE as VideoPrism, VideoCLIP-XL, and Perception Encoder.Source: §A.8 · pp. 18–19, Table 13 · p. 19, Table 15 · p. 20

A.9 SENSITIVITY TO THE CANDIDATE-POOL SIZE Nc

  • Adding candidates does not uniformly improve R@1: MSR-VTT/VP declines at 50 candidates then partly recovers, while ActivityNet/VP improves through 100.Figure 3 covers T2V R@1 for three datasets and two first-stage retrievers over 20 to 200 candidates; non-20 entries are single-run evaluations.Source: §A.9 · p. 21, Figure 3 · p. 21

A.10 16-TOKEN PRECISION DETAILS

  • At 16 cached tokens, FP8 leaves T2V and V2T recall effectively unchanged from BF16, while FP4 keeps T2V R@5 and R@10 within 0.1 percentage point.The reported precision reductions have a small effect on recall at this cache budget.Source: §A.10 · p. 21, Table 16 · p. 21

A.11 16-TOKEN FUTURE-DELTA DETAILS

  • At one cached token per sampled frame, Ldelta improves every reported cutoff in both directions, with the largest gain at T2V R@1.At 16 tokens, T2V gains are +1.9 R@1, +0.8 R@5, and +1.1 R@10 percentage points; V2T gains range from +0.5 R@1 to +0.8 R@10.Source: §A.11 · p. 21, Table 17 · p. 22

A.12 AUXILIARY PREDICTION TARGETS

  • With 64 cached tokens, the delta target exceeds current-frame reconstruction by 0.4 T2V and 0.7 V2T mean R@1 points, and the absolute-future target by 1.8 and 0.8 points, respectively.The absolute-future-feature target has 1.3 points higher T2V R@5 than the delta target.Source: §A.12 · p. 22, Table 18 · p. 22

A.13 TRAINING-LOSS STACK ABLATION

The appendix tests incremental training objectives, prediction horizons, and reranker capacity on MSR-VTT 1ka. The comparisons show that gains depend on the objective and horizon, while a larger reranker does not improve R@1 under the tested training schedule.

A.13 TRAINING-LOSS STACK ABLATION

  • +2.7 T2V R@1 from VTM over the VideoPrism prior, followed by conditional changes of −0.1/+0.8 from MLM, +1.3/+2.2 from VTC, and +0.6/+1.0 from Ldelta in T2V/V2T.Adding Ldelta lowers T2V R@5 by 0.9 points; each increment is conditional on previously included losses.Source: §A.13 · p. 22, Table 19 · p. 23

A.14 FUTURE-DELTA LOSS HORIZON SWEEP

  • Among tested horizons, h=3 improves R@1 over the no-auxiliary reference in both directions, while h=2 and h=4 do not improve both.Relative to 54.0/53.6 T2V/V2T R@1, h=2 gives 53.8/53.0 and h=4 gives 54.4/53.0; h=3 was selected using three-fold cross-validation on the training set.Source: §A.14 · p. 23, Table 20 · p. 23

A.15 RERANKER TEXT-ENCODER CONTROL

  • BERT-base adds 77M parameters, a 3.3× increase over the 33M reranker, but lowers R@1 by 1.7 T2V and 0.6 V2T points.It raises T2V R@5 and R@10 by 0.6 and 0.1 points, respectively; increasing capacity under this training schedule does not improve R@1.Source: §A.15 · pp. 23–24, Table 21 · p. 23

B COMPARISON WITH PUBLISHED SYSTEMS

The comparison places MSR-VTT T2V R@1 alongside online parameter counts for published systems, with counts tied to specified query-time modules. The paper cautions that parameter footprint alone does not determine execution time.

B COMPARISON WITH PUBLISHED SYSTEMS

  • The VEDJE total includes the first-stage text encoder and approximately 33M joint reranker; its visual backbone, compressor, and training-only predictor run during indexing or training.Reported totals are 143M for VideoPrism, 157M for VideoCLIP-XL, and 387M for PE-Core-B/16.Source: B COMPARISON WITH PUBLISHED SYSTEMS · pp. 24–25, Table 22 · p. 24
  • VEDJE reaches 59.8 R@1 at approximately 157M online parameters, close to LamRA’s reproduced 59.7 R@1, while CaRe-DPO reaches 64.1 and its base decoder alone has about 49 times VEDJE’s parameter count.The comparison counts the VideoChat-Flash-7B base decoder for MLLM systems; complete pipelines also use first-stage query encoding and fine-tuning components.Source: Table 22 · p. 24, B COMPARISON WITH PUBLISHED SYSTEMS · p. 25
  • Parameter footprint is not execution time: operation count, sequence length, candidate count, hardware, and cache access also affect latency.The paper reports measured VEDJE query latency separately, including text encoding, search, cache access, and scoring.Source: B COMPARISON WITH PUBLISHED SYSTEMS · pp. 24–25
Loading 2610.11850v1…