PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Similarity: Foundation Models as an Efficient Backbone for Training-Free Composed Video Retrieval

Dmitry Demidov, Muhammad Zaigham Zaheer, Omkar Thawakar, Abdelrahman Mohamed Shaker, Rao Anwer

arXiv:2609.10008v1cs.CV

TL;DR

CoVR needs efficient reusable search without losing transient actions, state changes, and subtle constraints that require fine-grained temporal reasoning. The paper introduces CoVRAGE, a training-free cascade that assigns frozen foundation models complementary roles and escalates inference with query difficulty. It achieves 89.55 R@1 on Dense-WebVid-CoVR and 93.43 R@1 on CoVR-R under complete target-gallery evaluation.

  • Problem

    CoVR must balance reusable, efficient gallery search against fine-grained temporal evidence that compact embeddings may miss.

  • Method

    CoVRAGE uses reusable gallery encoding, confidence-gated reranking and expansion, conditional target-description generation, and strict multimodal verification.

  • Results

    89.55 R@1 on Dense-WebVid-CoVR and 93.43 R@1 on CoVR-R are the highest displayed training-free values under the evaluated protocols.

  • Takeaways & Limitations

    Adaptive allocation combines gallery-scale retrieval with the fine-grained reasoning needed for difficult edits without task-specific training.

  • Takeaways & Limitations

    Fixed gates and the bounded candidate ceiling remain a hard recall boundary because targets omitted upstream are unreachable by verification.

Abstract

from arXiv · show

Composed video retrieval (CoVR) searches a gallery for the target video that realizes a natural-language modification of a source clip. However, at gallery scale, this creates a fundamental tension: compact embeddings enable efficient, reusable search but can miss the transient actions, state changes, and subtle constraints that demand fine-grained video reasoning, whereas applying large multimodal models uniformly sacrifices scalability. To address these limitations, we propose that frozen foundation models should instead occupy complementary roles, with inference depth adapted to query difficulty. Based on this premise, we introduce \methodname{}, a framework for training-free \methodexpansion{}. Specifically, a composed-query embedding first searches reusable video-only gallery representations; uncertain queries undergo bounded reranking and candidate expansion; ambiguous edits trigger target-description generation; and only close leading candidates reach multimodal verification. To support these roles, frame selection, spatial resolution, and time cues are adapted to each stage. Across complete target-gallery evaluations, our method reaches state-of-the-art performance among training-free approaches, with 89.55 and 93.43 R@1 on Dense-WebVid-CoVR and CoVR-R, respectively (with more than +35\% and +25\% absolute margins to the closest counterpart). These results show that adaptively orchestrating foundation-model capabilities can combine scalable retrieval with fine-grained reasoning without task-specific training. The source code and all relevant guidelines are available on https://github.com/demidovd98/CoVRAGE.

1. Introduction

Composed video retrieval must preserve efficient gallery-scale search while resolving transient temporal evidence and subtle edit constraints. CoVRAGE addresses this tension with a confidence-adaptive, training-free cascade that escalates only ambiguous queries and achieves the strongest reported training-free results on the evaluated benchmarks.

  • CoVR requires temporal reasoning because decisive actions, order, duration, and fleeting state changes can distinguish otherwise similar videos.
  • Compact embeddings enable reusable search but can lose fine-grained visual information needed for difficult composed edits.
  • CoVRAGE assigns frozen foundation models specialized roles and adapts inference depth according to query confidence.
  • The cascade combines reusable gallery encoding, compact retrieval, confidence-gated reranking, conditional target-description generation, and strict verification.
  • 89.55 R@1 on Dense-WebVid-CoVR and 93.43 R@1 on CoVR-R are the highest displayed training-free values under the evaluated protocols.

2. Related Work

Related work spans composed image and video retrieval, specialized supervised CoVR systems, reusable multimodal representations, and increasingly adaptive reasoning pipelines. This paper positions its contribution as a joint training-free design of reusable search, confidence-controlled refinement, and temporal evidence under complete target-gallery evaluation.

  • Composed image retrieval progressed from explicit visual-textual composition and free-form benchmarks toward zero-shot and multi-positive retrieval.
  • CoVR introduced retrieval of a target video from a source visual and an edit, while video adds sparse temporal evidence and query-dependent multi-frame cost.
  • Direct CoVR methods use task-specific supervision, temporal modeling, alignment objectives, prompt modules, and other specialized components.
  • Instruction-tuned multimodal embedders, dedicated rerankers, and universal retrieval systems support indexable representations and fine-grained candidate scoring.
  • Existing training-free pipelines explore reusable gallery representations, exhaustive judgments, fixed-depth reranking, and visual-guided reasoning.
  • The paper evaluates representation, capacity, routing, candidate depth, temporal evidence, resolution, and prompting within one training-free pipeline and target-gallery boundary.

3. Method

The method formulates composed video retrieval as ranking the gallery target that realizes a free-form modification relative to a source video. Its stated objective is to place that target first without task-specific parameter updates.

  • A composed query consists of a source video and free-form modification, while the gallery contains candidate target videos.

A confidence-gated training-free backbone

The confidence-gated backbone combines reusable coarse retrieval with progressively more selective candidate reasoning. It allocates reranking, description generation, verification, and temporal evidence only where uncertainty warrants deeper computation.

  • Reusable target representations support broad search, while harder queries receive deeper reasoning.
  • Gallery videos are encoded independently of queries, making their representations reusable across searches.
  • The cascade searches the complete gallery with composed-query embeddings before routing uncertain queries through bounded refinement stages.
  • Coarse retrieval uses uniform source and target views to prioritize gallery-wide recall, while later models process only a small leading pool.
  • Uncertain queries receive candidate-wise relevance scoring, and candidate pools expand only while the strongest score remains weak.
  • Short or implicit edits trigger a candidate-independent target description used only for reranking, with failed generation retaining the raw edit.
  • Near-tied candidates undergo strict full-match verification within a contiguous leading ambiguity cluster, while separated or implausible candidates are excluded.
  • Novelty-weighted, query-independent frame selection reallocates later-stage views toward temporal change while preserving reusable gallery indexing.

4. Experiments and Analysis

The experiments evaluate CoVRAGE under complete target-gallery protocols, stage-wise ablations, and qualitative failure cases. Results show strong cross-dataset retrieval and complementary gains from joint composition, candidate expansion, decomposition, and temporal verification.

  • Cross-Dataset Performance: 89.55 and 93.43 R@1 are achieved on Dense-WebVid-CoVR and CoVR-R, respectively, the highest displayed training-free values under the evaluated protocols.WebVid-CoVR reaches 54.58 R@1; the complete evaluations use target galleries with source self-masking.
  • Cross-Dataset Performance: 37.95 points separate CoVRAGE from the highest other source-reported training-free R@1 on CoVR-R.Against MoRe on Dense-WebVid-CoVR, CoVRAGE reaches 89.55 versus 49.6 R@1, 95.15 versus 67.0 R@5, and 96.40 versus 77.9 R@10.
  • Qualitative Results: Timestamp-aware verification rejects a plausible passenger clip and promotes the train-window target when appearance and setting must be jointly satisfied.The combined timestamp overlay and explanatory prompt provide the correction; the evidence does not isolate either element’s contribution.
  • Query Strategy Analysis: 14.50 R@1 points over video-only retrieval and 4.20 over the strongest late-fusion baseline result from joint source-video/edit encoding.Joint encoding also gives the best MeanR3, 80.73, supporting source-grounded composition.
  • Score-Based Candidate Pool Expansion: R@1 rises from 86.10 to 89.50 when uncertainty-triggered candidate expansion is enabled.MeanR3 increases from 90.97 to 94.97 while R@50 remains unchanged, indicating deeper coverage for low-confidence queries without changing broad recall.
  • Text Decomposition for Reranker: 4.10 of the 4.80-point R@1 gain is retained by selective target decomposition at 26.50% activation.The final interface confines generated text to reranking because embedding-side injection has mixed effects.

5. Conclusion and Discussion

The fully training-free CoVR cascade combines reusable gallery search with selectively routed reranking, decomposition, and verification, achieving the highest reported results among compared training-free methods under the stated protocols. Broader analysis identifies complementary corrections from temporal conditioning, joint composition, candidate depth, and semantic enrichment, while routing calibration and adaptive candidate budgets remain next steps.

  • 89.55 R@1 on Dense-WebVid-CoVR and 93.43 R@1 on CoVR-R are the highest results among compared training-free methods under these protocols.The evaluation uses complete cross-dataset target-gallery settings described in the conclusion.
  • The cascade combines reusable gallery search with selectively routed reranking, decomposition, and verification under stage-specific temporal evidence.
  • Qualitative and ablation evidence identifies complementary corrections from temporal conditioning, joint composition, candidate depth, and semantic enrichment.
  • Routing calibration and adaptive candidate budgets remain key next steps.

A. Additional Method Details

The additional method details specify a staged execution pipeline that reuses gallery embeddings, adapts frame evidence to clip duration and novelty, and routes uncertain queries through bounded refinement and verification. Explicit thresholds, fallback invariants, and candidate limits constrain each downstream stage.

  • Confidence-controlled refinement: Reranking is bypassed only when e1−e2 > 0.25; otherwise, expansion continues while the best relevance score remains strictly below 0.70, up to 50 scored items.Equality at both thresholds enters or stops the corresponding stage as specified.
  • Reusable gallery construction: Each distinct target is embedded once into a reusable video-only gallery matrix after validation, deduplication, and query-independent target selection.
  • Online execution: The initial five embedding-ranked items seed later stages, while a sufficiently separated leader returns the coarse ranking directly.
  • Stage-aware frame allocation: At most B = 15 selected frames reach each downstream video input, while reranking, decomposition, and verification use duration-adaptive novelty allocation from low-resolution probes.Short clips or failed decodes may provide fewer frames; probe computation is accounted for separately.
  • Stage-aware frame allocation: Novelty-weighted sampling distributes frames over novelty mass rather than selecting only the largest changes, with a uniform fallback when novelty is zero.
  • Embedding inputs: The composed-query encoder uses the source and raw edit, whereas the gallery encoder uses only the target video; both outputs are L2-normalized for cosine similarity.
  • Confidence-controlled refinement: The candidate scorer omits the source video, using the raw edit or one reused target description with each candidate video.
  • Multimodal verification: Verification evaluates only the reranked top ten using the source, raw edit, and one candidate, requiring every modified condition to be satisfied.Unconstrained attributes need not match, while missing, contradicted, or uncertain conditions yield a negative verdict.

B.2.1. Cross-Dataset Performance

CoVRAGE improves retrieval most clearly on denser, manually verified benchmarks, while its gains arise from confidence-controlled specialization across reranking, decomposition, and candidate expansion. The cascade also exposes boundaries from ambiguous annotations, upstream candidate omission, and single-positive evaluation.

  • Cross-dataset results: 89.55/95.15/96.40/97.26 Dense-WebVid-CoVR recall yields 94.59 Avg., with the largest improvement at R@1.R@50 is already close to saturation.
  • Cross-dataset results: 93.43/94.27/94.61/94.95 CoVR-R recall exceeds other directly compared training-free values at every cutoff.The largest margin is again at R@1, improving early discrimination and coverage without repeating gallery-wide reasoning.
  • Evaluation boundaries: WebVid-CoVR retains short, underspecified edits, whereas Dense-WebVid-CoVR uses fully manually verified and corrected modifications.CoVRAGE is less than one point below the best complete training-free average on WebVid-CoVR but exceeds MoRe by 39.95, 28.15, and 18.50 points at R@1, R@5, and R@10 on Dense-WebVid-CoVR.
  • Where gains arise: Candidate reranking and routed decomposition produce the dominant R@1 gains, while verification adds 1.33, 0.98, and 0.11 points across WebVid-CoVR, Dense-WebVid-CoVR, and CoVR-R.R@50 remains unchanged because later stages reorder a bounded head while the embedding tail preserves coverage.
  • Evaluation boundaries: Sparse edits can leave the target outside the bounded candidate pool, making later reasoning unable to recover it; denser edits can restore upstream coverage.A separate campfire case shows that single-positive evaluation can mark a plausible near-duplicate as incorrect.
  • Reranking and expansion: Confidence-triggered candidate-pool growth improves R@1 from 86.10 to 89.50 and MeanR3 from 90.97 to 94.97.The matched combined-query study attributes the gain to pool depth rather than the text interface.
  • Reranking and decomposition: Target-oriented decomposition raises Dense-WebVid-CoVR R@1 from 89.50 to 94.30, while ambiguity routing activates for 26.50% of queries and retains most of the always-on gain.Using generated descriptions directly in embedding has mixed cross-dataset effects, so the final interface confines decomposition to reranking.

B.4.4. Verifier Stage

The verifier repairs residual candidate-order errors through strict multimodal comparison, but confidence gating limits this expensive operation to leading ambiguity clusters. Temporal and dynamic evidence improve matched results, while the reported temporal overlay effect is attributable only to the combined package.

  • Verifier stage: Ungated verification improves R@1 from 93.60 to 94.20, confirming that final multimodal comparison can repair residual candidate-order errors.
  • Verifier stage: Relative-score and score-floor gating reduces verifier work by more than 90% with only a small accuracy change.The scan is restricted to a leading ambiguity cluster rather than every candidate set.
  • Verifier stage: Strict full-condition checking raises gated R@1 from 94.00 to 94.50 under the fixed source-edit-candidate interface.The result supports a strict, confidence-gated, top-down verifier.
  • Frame sampling: Novelty-weighted allocation improves the matched Dense-WebVid-CoVR result from 94.40 to 95.00 R@1 and improves every reported recall cutoff.The selected profile keeps gallery-wide embedding uniform and applies dynamic views to candidate and reasoning stages.
  • Frame sampling: Short, medium, and long clips are probed at 8, 4, and 2 frames per second, while later-stage gains are not uniform across duration groups.Shorter groups benefit, whereas long clips expose heterogeneity and small-sample uncertainty in the long-duration tail.
  • Temporal cues: The combined timestamp-overlay and explanatory-prompt package changes R@1 from 94.70 to 95.00 and MRR from 0.9652 to 0.9663, while R@50 is unchanged.The result does not isolate overlay rendering from prompt wording.

B.5. Efficiency and Additional Analysis

The efficiency study measures conditional online inference and shows that workload varies substantially with edit style, while reusable preparation and model choices shape the serving boundary. The reported timings are measured within a controlled evaluation setup rather than as hardware-independent speed comparisons.

  • Online inference cost: 69.751, 38.630, and 17.842 seconds are the mean online busy times per query on WebVid-CoVR, Dense-WebVid-CoVR, and CoVR-R, respectively.The corresponding compute is 76.264, 50.514, and 23.227 model inferences per query; gate skips count as zero calls.
  • Routing workload: WebVid-CoVR activates expansion, decomposition, and verification for 75.4%, 99.6%, and 16.1% of queries, versus 17.4%, 11.1%, and 3.6% on CoVR-R.Dense-WebVid-CoVR has intermediate activation rates of 44.2%, 26.1%, and 6.5%, showing that edit distributions drive cost differences.
  • Stage breakdown: 47.898, 31.848, and 13.741 busy seconds per query are attributed to reranking across WebVid-CoVR, Dense-WebVid-CoVR, and CoVR-R.Query embedding remains nearly constant at 1.434–1.483 seconds and one call per query, while conditional generation scales with activation rates.
  • Measurement boundary: 5,330.73, 3,233.29, and 2,012.57 seconds are evaluation-routine wall times, but they are not hardware-independent latency or cross-system speed comparisons.The measurements reflect a serving environment with 16-worker overlap and retries; model-call counts are workload measures, not FLOPs or monetary cost.
  • Model roles: The implementation uses Qwen3-VL-Embedding-2B for composed-query retrieval, Qwen3-VL-Reranker-8B for candidate scoring, and Qwen3.5-9B for decomposition and verification.The generative roles use separate prompts, layouts, parsers, resolutions, and reusable prediction records.

C.1. Caching, Presampling, Concurrency, and Reproducibility

The configuration separates reusable representations and temporal preparation from conditional online stages, with explicit caching, frame-policy, routing, and execution rules. Controlled tables document workload, frame-policy, temporal, cross-dataset, and routing behavior under these settings.

  • Caching: Four independent reuse families cache embeddings, reranker predictions, decomposer and verifier predictions, and temporal-probe selections using structured keys.Keys include model or stage identity, prompt or instruction, content, frame signatures, resolution, timestamps, and selector metadata where applicable.
  • Reproducibility: Timed evaluations redirect or disable persistent stores while retaining safe within-invocation reuse, preventing earlier runs from shrinking the measured cold boundary.Writes are atomic and malformed entries become misses.
  • Execution: The final configuration resolves temporal probes and selected indices before inference but keeps complete frame bundles lazy; repeated candidate calls reuse candidate frames.Independent items and queries run concurrently, while expansion rounds and verifier scans remain sequential within a query.
  • Frame policy: The dynamic frame package improves the matched aggregate result at every reported recall cutoff under a fixed 15-frame output budget per video input.Probe work is accounted separately from selected output frames.
  • Temporal evidence: Temporal timestamp rows jointly change decomposer overlays, verifier overlays, and the timestamp note, so they are not component-wise causal estimates.The timestamp overlay supplies elapsed seconds as an absolute temporal cue for irregularly spaced samples.
  • Controlled profiling: The controlled profiles report online busy time, issued model calls, reusable preparation, and query-independent temporal selection for size-matched cohorts.Table S13 counts one low-resolution probe call per inspected frame and identifies unique clip-stage selections and gallery embeddings as reusable.
  • Configuration boundary: The 15-frame setting is an upper bound per video input, while short or unreadable clips may yield fewer frames and the selector may inspect additional low-resolution probes.Verification receives separate source and candidate inputs.

D. Limitations and Future Directions

The study covers stage-wise behavior comprehensively but leaves important scope and calibration questions open. In particular, fixed routing and query-independent selection impose boundaries that can affect recall and temporal evidence coverage.

  • Scope: The study is not exhaustive over every foundation-model family, and stochastic generative ablations are primarily single evaluations.The authors identify broader model-family coverage and stronger stochastic evaluation as open directions.
  • Temporal selection: Query-independent dynamic selection can under-allocate slowly evolving evidence, motivating training-free query-aware selection.This limitation concerns how temporal evidence is allocated before the query-specific reasoning stages.
  • Candidate recall: Fixed gates and the bounded candidate ceiling form a hard recall boundary because targets omitted upstream are unreachable by verification.The paper also notes that strict verification can reject a correct near-tied candidate.
  • Evaluation: Target-only galleries and single-positive annotations under-measure multiple-valid-answer uncertainty.The authors point to the campfire case as an example and propose multiple-valid-target evaluation.
Loading 2609.10008v1…