PaperlayerView exact revision on arXiv

Source-linked AI summary

Allocate Before You Embed: Adaptive Visual Input Allocation for Video Embeddings

Song Jin, Zhongtao Jiang, Chenglei Shen, Huanxuan Liao, Haozhe Chi, Zhiwei Wang, Kun Xu, Yong Liu

arXiv:2609.01778v1cs.CV

TL;DR

Large-scale video retrieval must encode long videos under limited visual-input budgets, but fixed-frame sampling can miss brief or unevenly distributed evidence. AllocEmbed reallocates the budget across denser frames using a lightweight pre-embedding allocator, while RDPO learns allocation from retrieval feedback. Across MMEB-V2 and LongRet, it improves aggregate retrieval under matched budgets and transfers across backbones without retraining.

  • Problem

    Fixed-frame video inputs restrict temporal coverage and treat different frames and visual evidence as equally valuable under costly visual processing.

  • Method

    AllocEmbed predicts content- and text-conditioned frame resolutions before a frozen embedding backbone, while RDPO learns allocations from ranking feedback and efficiency incentives.

  • Results

    AllocEmbed improves aggregate retrieval under matched visual-input budgets across MMEB-V2 and LongRet, with LongRet gains of 5.69–9.09 points and transfer across backbones without retraining.

  • Takeaways & Limitations

    Input-side visual allocation provides a practical way to improve video retrieval while preserving existing embedding backbones and downstream retrieval pipelines.

Abstract

from arXiv · show

Large-scale video retrieval requires embedding models to encode long and diverse videos under tight visual-input and inference budgets. Existing methods typically sample a small, fixed set of frames at their original resolution, limiting temporal coverage and ignoring frame importance. Our empirical analysis shows that expanding temporal coverage improves retrieval even under a fixed visual-input budget. Gains are larger when the original per-frame resolution is preserved, highlighting the complementary roles of temporal coverage and spatial fidelity. Motivated by this finding, we propose AllocEmbed, an allocate-then-embed framework that reallocates a fixed visual-input budget across more frames. A lightweight allocator uses low-cost previews to assign frame-wise resolutions before the embedding backbone, preserving more detail where it most benefits retrieval while reducing visual cost elsewhere. We further introduce Retrieval-Driven Policy Optimization (RDPO), which learns the allocator directly from retrieval feedback using a rank-validated similarity gap and a confidence-guided efficiency incentive. Operating entirely before the backbone, AllocEmbed integrates with existing retrieval systems without modifying the embedding model or downstream pipeline. Experiments on the MMEB-V2 V-QA and V-RET tasks and our LongRet benchmark show that AllocEmbed achieves the best overall retrieval performance among the evaluated budget-matched methods and transfers across embedding backbones. Our code is publicly available at https://github.com/jinsong8/AllocEmbed.

1 Introduction

Video retrieval systems often use small fixed-frame inputs, limiting temporal coverage while treating visual evidence as equally valuable. AllocEmbed reallocates a fixed visual-input budget across denser frame sequences, and RDPO learns this allocation from retrieval feedback.

  • Fixed-frame sampling is cost-effective but can miss retrieval-relevant evidence because it assumes frames and visual evidence have equal value.
  • Temporal coverage and spatial fidelity provide complementary benefits, motivating adaptive allocation of a fixed budget across denser frame sequences.
  • AllocEmbed predicts frame-wise resolution scales from low-cost previews and task text, then embeds the resulting mixed-resolution input with a frozen backbone.
  • RDPO learns allocation policies directly from retrieval outcomes using a rank-validated similarity gap and a confidence-guided efficiency incentive.
  • 1.34–1.69 points: AllocEmbed’s average Hit@1 improvement across ten MMEB-V2 tasks over corresponding fixed-frame baselines.Across four LongRet subsets, gains increase to 5.69–9.09 points without exceeding each backbone’s fixed-frame visual-input cost.
  • AllocEmbed transfers from its training backbone to three additional embedding backbones without retraining.

2 Related Work

Prior video retrieval research includes dedicated video–text encoders, image–text models adapted through frame aggregation, and efficient processing methods. The trade-off between temporal coverage and spatial detail under a constrained budget remains underexplored.

  • Earlier video embedding approaches either pretrain dedicated video–text encoders or adapt image–text models through frame-level aggregation.
  • Figure 2 compares Full Resolution, Uniform Allocation, and AllocEmbed as sampled frame counts increase across ten MMEB-V2 tasks.
  • The trade-off between temporal coverage and per-frame spatial detail under constrained visual-input budgets remains underexplored.
  • Frame-selection, token compression, and content-adaptive resolution methods reduce visual processing costs in related video tasks.

3 Temporal Coverage and Spatial Fidelity

Experiments show that increasing frame counts improves retrieval when resolution is preserved and can also help under fixed visual budgets, though excessive downscaling eventually reduces the benefit. AllocEmbed responds by allocating resolution non-uniformly across denser sequences.

  • Temporal coverage: Across four backbones, denser sampling improves retrieval when original per-frame resolution is preserved.
  • Fixed-budget allocation: Under a fixed visual-input budget, moderately increasing uniformly resized frames improves retrieval across all four backbones.
  • Fixed-budget allocation: The fixed-budget benefit diminishes when aggressive downscaling causes spatial-detail loss to outweigh broader temporal coverage.
  • AllocEmbed motivation: Figure 3 depicts an allocator that predicts per-frame scales from video and task text before constructing mixed-resolution inputs for a frozen embedding backbone.
  • AllocEmbed motivation: AllocEmbed retains denser frame sequences while allocating resolution non-uniformly, avoiding uniform spatial-detail loss from global downscaling.

4 AllocEmbed

AllocEmbed allocates a fixed visual-input budget across densely sampled frames before embedding, using content- and text-conditioned resolutions to preserve useful spatial detail. RDPO trains this allocator from retrieval outcomes while keeping the embedding backbone and downstream retrieval pipeline unchanged.

  • Problem formulation: Standard pipelines sample few frames uniformly, which can miss brief or sparsely distributed retrieval-relevant evidence.
  • Problem formulation: AllocEmbed samples a denser candidate sequence and assigns each frame a scale controlling retained spatial information under a fixed budget.The allocation redistributes resources across frames rather than uniformly applying one resolution.
  • Allocate-then-embed design: A lightweight allocator maps candidate frames and task text to frame-wise scales, producing a mixed-resolution sequence for the embedding backbone.The allocator first predicts s = πθ(X, q), then scale-aware preprocessing resizes each frame accordingly.
  • Allocate-then-embed design: Larger scales preserve more spatial detail, while smaller scales reduce visual cost so more frames can fit within the same budget.The policy can assign most frames scales below 1 and can adjust the allowed scale range to control visual budget.
  • Retrieval-Driven Policy Optimization: RDPO compares multiple allocations for the same input using shared retrieval candidates and a rank-validated similarity gap derived from retrieval outcomes.Its group-relative normalization reduces sensitivity to differences in example difficulty.
  • Retrieval-Driven Policy Optimization: RDPO adds an efficiency incentive only after successful retrieval, rewarding lower average scale more strongly when the positive target has sufficient confidence.Unsuccessful allocations cannot benefit merely from using smaller scales, while successful low-gap allocations receive only weak efficiency incentives.

5 Experiments

AllocEmbed improves budget-matched video retrieval by reallocating visual input across denser frame sequences, with especially strong gains on long videos. Ablations and transfer experiments support the effectiveness of RDPO-guided allocation across models and budgets.

  • Main Results: AllocEmbed achieves the best budget-matched average with both 2B models while matching or reducing Base cost.This supports balancing broader temporal coverage with spatial fidelity through learned frame-wise resolution allocation.
  • Main Results: 5.69–9.09 Hit@1 points are gained over Base across four LongRet subsets at roughly one-third of Full Resolution’s cost.AllocEmbed is the strongest budget-matched method on every LongRet subset.
  • Ablation Studies: Full RDPO provides the best quality–cost balance, while retrieval success alone is too sparse and ungated similarity gaps increase visual cost.Combining success gating with graded similarity feedback supplies both a reliable success criterion and ranking information.
  • Generalization to Diverse Embedding Models: AllocEmbed improves both MMEB-V2 and LongRet averages across four embedding models spanning 2B to 8B parameters while matching or reducing Base cost.The allocator is trained only with VLM2Vec-V2-2B and transfers without further training.
  • Performance across Visual Input Budgets: AllocEmbed outperforms Base at every tested budget on both benchmark groups, reaching a 6.25 Hit@1-point LongRet gain at the largest budget.Base uses 8, 16, or 20 native-resolution frames, while AllocEmbed uses matched candidate pools of 24, 48, or 60 frames.
  • Allocation Analysis: Informative, high-quality frames receive larger scales, while blank, blurred, or transitional frames receive smaller scales.Examples include red-carpet, statuette, location-text, sharp-game-footage, and text-bearing frames.

6 Conclusion

The paper concludes that AllocEmbed allocates a constrained visual-input budget across denser frame sequences before embedding, while RDPO learns this allocation from retrieval outcomes. It improves aggregate retrieval under matched budgets and transfers across frozen backbones without retraining.

  • Conclusion: AllocEmbed balances temporal coverage and spatial fidelity by distributing a constrained visual-input budget across a denser frame sequence.It predicts frame resolutions from video content and task text before embedding, without architectural changes to existing VLM-based embedding models.
  • Conclusion: RDPO learns frame-wise allocation from retrieval outcomes using a rank-validated similarity gap and a confidence-guided efficiency incentive.The allocator operates before the embedding backbone.
  • Conclusion: Experiments show improved aggregate retrieval under matched visual-input budgets and transfer across 2B-to-8B backbones without retraining.The conclusion identifies input-side visual allocation as a practical approach for existing embedding backbones.

A.1 Implementation Details

Implementation freezes the preview extractor and embedding backbone, training only the frame-wise allocation module with RDPO. The allocator predicts continuous frame scales whose visual cost grows approximately quadratically with scale.

  • Training Configuration: Only the frame-wise allocation module is optimized while the SmolVLM2-256M feature extractor and VLM2Vec-V2-2B backbone remain frozen.Training uses the VLM2Vec-V2 video training set for 125 steps.
  • Training Configuration: Training uses eight NVIDIA H100 80GB GPUs, BF16 computation, a learning rate of 2 × 10−5, batch size 1,024, and K = 16 allocations per example.Each step yields 16,384 allocation-conditioned queries.
  • Scale Prediction: Each candidate frame receives a continuous scale in [0.2, 1.8], and visual input cost grows approximately quadratically because scale applies to both spatial dimensions.This scale range and cost relationship define the allocator’s resource-control mechanism.
  • RDPO Configuration: RDPO uses λeff = 0.4, γ = 0.45, and τ = 0.10, with a success gate preventing incorrect allocations from earning efficiency incentives solely by reducing scale.The training target pool combines 1,024 in-batch targets with 3,072 sampled text targets.

A.2 Benchmark and Baseline Details

The evaluation compares budget-matched frame-allocation strategies on MMEB-V2 and LongRet using fixed retrieval inputs and Hit@1. AllocEmbed predicts one allocation at inference and invokes the frozen backbone once after scale-aware preprocessing.

  • MMEB-V2: MMEB-V2 contains five video-retrieval and five video-question-answering tasks, evaluated with four embedding models.The models span VLM2Vec-V2-2B, Qwen3-VL-Embedding-2B, VLM2Vec-Qwen2VL-7B, and Qwen3-VL-Embedding-8B.
  • LongRet: LongRet retrieves complete source videos from localized or complete-video descriptions across LoVR and Vript corpora.Its Clip and Video subsets differ in query granularity and count while sharing targets within each corpus.
  • Baselines: Base embeds 8 uniformly sampled native-resolution frames, whereas other methods begin with 24 candidates and allocate or select frames differently.Full Resolution uses all 24 frames at Base resolution and is not budget matched.
  • Evaluation Protocol: Methods hold the input video, task text, retrieval targets, and metric fixed, reporting Hit@1 with realized visual input cost.All methods except Full Resolution are evaluated under matched budgets.
  • Inference: At inference, AllocEmbed predicts one allocation and invokes the frozen backbone once after scale-aware preprocessing.Group sampling with K = 16 is used only during training.

B Additional Analysis and Discussion

The analysis measures model-path inference overhead under paired, budget-capped conditions and finds that AllocEmbed substantially reduces total latency despite allocator cost. Reported timings exclude several preprocessing stages that remain available for systems optimization.

  • Measurement Setup: Paired LoVR-Video runs use the same videos, text, and 24 sampled frames under a 64K video-token cap.Experiments use VLM2Vec-V2.0 with BF16, batch size one, and a single NVIDIA H100 80GB GPU.
  • Latency Results: 149.59 ms of allocator overhead accompanies a backbone-latency reduction from 754.77 to 239.22 ms.AllocEmbed's scale predictions use two-frame chunks with scales in [0.2, 1.32], while the full-input baseline sends all 24 frames directly to the backbone.
  • Latency Results: 48.49% lower total model-path latency corresponds to a 1.941× speedup over full-input inference.The latency measurements exclude allocator preprocessing, scale-dependent resizing, video transforms, and backbone preprocessing.
  • Systems Considerations: Excluded preprocessing stages can run on the CPU and may be further optimized through parallelism and pipelining.The reported measurements therefore cover model-path latency rather than a fully optimized end-to-end implementation.

B.3 Complete LongRet Results

The complete LongRet results are reported at the subset level in Table 7, corresponding to Figure 4. The table compares uniform, content-aware, and full-resolution configurations under stated frame and cost conventions.

  • Complete Results: Table 7 reports complete subset-level LongRet results corresponding to Figure 4.The table reports Hit@1 (%) for four subsets and their mean, with absolute changes from Base in parentheses.
  • Evaluation Conventions: Base embeds 8 uniformly sampled frames, while the other budget-matched methods begin with 24 candidate frames.Content-Aware Selection embeds 8 frames; Avg. is the mean over four subsets.
  • Evaluation Conventions: Table 7 normalizes cost to Base at 1.000× and bolds the best budget-matched result in each column.Values are Hit@1 (%), and parentheses show absolute changes from Base.
  • Evaluation Conventions: Full Resolution processes all 24 frames at the Base per-frame resolution and is explicitly not budget matched.This configuration is marked with a dagger in the table.
Loading 2609.01778v1…