Source-linked AI summary
PEEK: Picking Essential frames via Efficient Knowledge distillation
Killian Steunou, Anas Filali Razzouki, Khalil Guetari, Mounîm A. El-Yacoubi, Yannis Tevissen
TL;DR
Video captioning must select informative frames from long, redundant videos, but adaptive selectors can be expensive. PEEK distills caption-conditioned rankings into a lightweight visual-only selector, performing best when only one or two frames are available and adding less overhead than competing adaptive methods.
Problem
Video understanding is expensive because long, redundant videos require extracting sparse relevant cues from frame sequences.
Method
PEEK distills caption-conditioned frame rankings from a teacher into a lightweight, query-free temporal scorer operating only on visual features.
Results
PEEK is the strongest query-free selector across both benchmarks, achieving the best CIDEr in all one-frame settings and seven of eight two-frame settings.
Takeaways & Limitations
PEEK is particularly effective when frame budgets are tight, offering a practical efficiency trade-off for video captioning.
Takeaways & Limitations
The teacher signal relies on ground-truth captions, tying learned relevance to reference-caption alignment rather than all visually meaningful events.
Abstract
from arXiv · showhide
Video-language models can process only a limited number of frames, making frame selection a key bottleneck for efficient video captioning. Most captioning pipelines still rely on uniform sampling, which is computationally cheap but agnostic to visual content. Adaptive frame sampling has recently emerged as a promising approach for selecting the most informative frames from a video; however, existing methods remain computationally expensive. We introduce PEEK, an efficient dynamic frame sampling method that distills caption-conditioned frame relevance rankings from a stronger teacher model into a lightweight temporal model that operates only on visual content. We find that, overall, on ActivityNet Captions and MSR-VTT, our method outperforms state-of-the-art methods across all evaluated downstream vision language models, especially when only one or two frames are selected for captioning, obtaining the best CIDEr for most frame budgets. On ActivityNet Captions, PEEK is particularly strong, winning 14 out of 16 configurations. Zero-shot evaluation on MSR-VTT shows that our model transfers best at low frame budgets, while results at four and eight frames are more mixed as temporal coverage and visual diversity become increasingly competitive. Compared with recent adaptive baselines, PEEK is both more accurate in the low-budget regime and more efficient: it adds only $5.2\%$ to the captioning time, compared with $65.4\%$ for CSTA and $211.9\%$ for MaxInfo. We release our code and pre-trained checkpoint at https://github.com/momentslab/peek.
1 Introduction
PEEK addresses the cost and unreliability of selecting sparse, decisive video frames by distilling caption-conditioned relevance rankings into an efficient selector that uses visual features alone. It is designed to improve low-frame video captioning while reducing selection cost relative to content-aware baselines.
- Motivation: Video understanding is expensive because videos are long and redundant, while decisive visual cues may be sparse and absent from a limited frame budget.Vision-language models often receive only a glimpse of a video rather than a guarantee that the relevant cue is included.
- Motivation: Uniform sampling remains the default because it is deterministic, model-free, and often effective, despite being agnostic to visual content.It partitions videos into equal temporal segments and keeps one frame from each.
- Method: PEEK distills dense caption-conditioned frame rankings from a frozen vision-language teacher into a lightweight temporal Transformer that predicts relevance from visual embeddings alone.The teacher provides supervision offline, while the deployed selector is query-free and caption-agnostic.
- Method: Caption-conditioned frame scoring is introduced as an Oracle diagnostic for measuring the value of semantic frame relevance in video captioning.This relevance signal is used for supervision rather than as an inference-time query.
- Evaluation: On ActivityNet Captions and MSR-VTT, PEEK is evaluated with four downstream vision-language models, showing consistent low-frame gains and lower selection cost than recent content-aware baselines.The claimed benefits span both captioning performance and computational efficiency.
2 Related Work
Related work treats frame selection as a central design choice under fixed visual-token budgets, spanning uniform sampling, training-free adaptive methods, learned selectors, and text-conditioned scoring. PEEK is positioned as a caption-oriented, lightweight selector that distills caption-frame relevance rankings rather than directly optimizing diversity or using a large vision-language model during selection.
- Sampling strategies: Uniform sampling remains widely used because it is deterministic, model-free, and cheap, while serving as a strong baseline.Frame-sampling choices can substantially affect video question-answering results in controlled benchmarks for small VLMs.
- Training-free frame selection: Training-free adaptive methods select keyframes using informativeness, diversity, temporal coverage, or text relevance.MaxInfo maximizes the geometric volume spanned by frame embeddings, while other methods combine text relevance with visual coverage.
- Learned frame selectors: Learned selectors frame selection as a trainable decision process, but they are often tied to the downstream model or input query.Examples include Frame-Voyager, pseudo-label-based multimodal selectors, VideoBrain’s adaptive acquisition process, and CSTA’s learned importance scores for summarization.
- Frame selection for video captioning: Video captioning is especially constrained because only visual input is available when frames are selected, making direct use of text-aware selectors difficult.PickNet uses reinforcement learning and task-specific rewards, while LFS is identified as more recent learnable work for captioning.
- Positioning of PEEK: PEEK is closest to PickNet and LFS but instead distills caption-frame relevance rankings, and unlike MaxInfo it learns a caption-oriented relevance prior rather than directly optimizing visual diversity.Its positioning also emphasizes avoiding a large vision-language model during selection and reducing deployment cost.
3 Method
PEEK learns a query-free temporal frame selector by distilling caption-conditioned relevance rankings from a strong teacher into a lightweight visual-only temporal scorer. At inference, it selects high-scoring frames while preserving temporal coverage for downstream captioning.
- Two-stage distillation: PEEK uses a two-stage framework: a text-conditioned teacher scores candidate frames against ground-truth captions, then a lightweight temporal scorer imitates the induced ranking without caption access.The student scores frames from unseen videos before caption generation.
- Teacher supervision: Training subsamples T candidate frames from each annotated temporal segment and retains only scalar teacher scores as supervision.Teacher scores are min–max rescaled to targets in [0,1], preserving the teacher’s internal ordering.
- Student scorer: The student uses frozen 512-dimensional MobileCLIP2 inputs, while SigLIP 2 supplies supervision targets; its temporal scorer has about 1.7M trainable parameters and 13.1M total parameters.SigLIP 2 is used only to produce supervision targets, excluding frozen encoder parameters from the trainable count.
- Ranking objective: ListMLE optimizes the probability of the teacher-induced ranking rather than pointwise values or pairwise comparisons, aligning training with frame selection.The ranking is sorted by decreasing teacher target, and the student outputs are treated as Plackett–Luce utilities.
- Frame selection: At test time, PEEK partitions a segment into k temporal sub-segments and selects the highest-scoring frame from each, combining local content selection with temporal coverage.For k = 1, this becomes selecting the single highest-scoring frame; selected frames are temporally ordered before captioning.
4 Experiments · 4.1 Data
The experiments train and evaluate PEEK primarily on ActivityNet Captions, then test zero-shot transfer to MSR-VTT, which differs in clip structure and caption distribution. The datasets therefore assess both in-domain event-level captioning and cross-dataset clip-level generalization.
- 4 Experiments: In a bagpipe-playing ActivityNet segment, PEEK and its SigLIP2 teacher select frames that reveal the instrument, whereas the uniform center frame misses it.The captions shown for the selected frames are generated by Qwen2.5-VL-3B.
- 4.1 Data: PEEK is trained on ActivityNet Captions using the official splits, with all reported metrics computed on its test set.MSR-VTT is additionally used for zero-shot transfer evaluation.
- 4.1 Data: ActivityNet Captions contains untrimmed YouTube videos with multiple temporally localized descriptions for densely annotated events.Each video averages 3–4 overlapping or sequential events and typically lasts about two minutes.
- 4.1 Data: Each annotated ActivityNet Captions event is described by a free-form English sentence.This supports event-level captioning over localized temporal segments.
- 4.1 Data: MSR-VTT consists of short web clips paired with 20 crowd-sourced English captions per clip.Unlike ActivityNet Captions, its captions describe the entire clip rather than localized events.
- 4.1 Data: MSR-VTT is used exclusively for evaluation to test whether training on ActivityNet Captions generalizes zero-shot to a different caption distribution.Each test video contributes one segment spanning the full clip.
4.2 Training and Evaluation
PEEK is trained on temporally grounded ANC clips using only visual and temporal features, then evaluated by selecting 1, 2, 4, or 8 frames for downstream video-language models on ANC and MSR-VTT. The evaluation compares PEEK with Oracle, Uniform, Random, MaxInfo, and CSTA under fixed candidates, parameters, and seeds.
- Training: PEEK is trained on ANC segments decoded at 2 fps, with long sequences capped and shorter ones zero-padded using attention masks.Stage 2 receives only visual features and temporal information, never captions, sentence boundaries, or external metadata.
- Evaluation: The selector is evaluated on ANC and MSR-VTT by choosing k ∈ {1,2,4,8} frames for downstream video-language models.PEEK is the ANC-trained student model using stratified argmax selection.
- Baselines: The comparison includes Oracle, Uniform, Random, MaxInfo, and CSTA as training-free frame-selection baselines.Oracle estimates an approximate upper bound; MaxInfo uses CLIP embeddings, while CSTA scores frames before selecting one per temporal sub-segment.
- Evaluation protocol: All methods use the same candidate frames, while downstream VLM parameters and random seeds are fixed for fair comparison.PickNet and LFS are excluded because official implementations or pretrained checkpoints were unavailable at submission.
- Captioning setup: Captions are generated from the k selected frames in temporal order as a single multi-image input followed by a one-sentence prompt.The evaluated VLMs are SmolVLM2-2.2B-Instruct, Qwen2.5-VL-3B, Qwen3.5-4B, and Qwen2.5-VL-7B.
4.3 Results
PEEK is the strongest query-free selector on ActivityNet Captions, especially at one- and two-frame budgets, and transfers best to zero-shot MSR-VTT at one frame. Its advantage narrows at larger budgets, where temporal coverage, diversity, and captioner-specific effects produce mixed results.
- ActivityNet Captions: 14 of 16 ActivityNet Captions model/budget settings have PEEK achieving the best CIDEr among query-free selectors.At k=1, PEEK improves over the strongest query-free baseline by 1.74–3.00 CIDEr points across the four evaluated VLMs.
- ActivityNet Captions: +1.74 CIDEr points is PEEK’s smallest reported k=1 ActivityNet gain, while the largest is +3.00 points.The gains are +1.74 for SmolVLM2-2.2B, +2.34 for Qwen2.5-VL-3B, +2.18 for Qwen3.5-4B, and +3.00 for Qwen2.5-VL-7B.
- ActivityNet Captions: At k=4 and k=8 on ActivityNet Captions, PEEK is best in three of four CIDEr settings at each budget, with small reversals against MaxInfo or Uniform.Uniform exceeds PEEK by 0.11 CIDEr points for Qwen3.5-4B at k=8.
- Zero-shot MSR-VTT: +2.68 CIDEr points is PEEK’s largest reported zero-shot MSR-VTT k=1 gain, and PEEK is best for all four downstream VLMs and all reported metrics.The remaining CIDEr gains are +1.46 for SmolVLM2-2.2B, +2.26 for Qwen3.5-4B, and +1.25 for Qwen2.5-VL-7B.
- Zero-shot MSR-VTT: At k=2 on MSR-VTT, PEEK is best in CIDEr for three of four VLMs, but at k=4 and k=8 Uniform, Random, or MaxInfo often lead.The sole k=2 CIDEr exception is Qwen2.5-VL-3B, where Random is higher by 0.19 CIDEr points; additional frames can also reduce CIDEr for some captioners.
4.4 Efficiency
PEEK substantially reduces frame-selection and end-to-end captioning overhead compared with content-aware baselines. Its low computational cost supports practical, query-free frame selection.
- Selection efficiency: 0.36s per segment is PEEK’s selection time, compared with 4.52s for CSTA and 14.62s for MaxInfo on ActivityNet Captions.PEEK processed all 17,505 segments in 1h44m of GPU time, versus 21h58m for CSTA and 71h04m for MaxInfo.
- End-to-end efficiency: 5.2% is PEEK’s increase in total GPU time over Uniform, compared with 65.4% for CSTA, 211.9% for MaxInfo, and 29.4% for the Oracle.These measurements reuse frame scores across the full k ∈{1,2,4,8} captioning pipeline.
- Practical value: PEEK remains query-free and lightweight while recovering part of the Oracle’s caption-relevance signal.The paper identifies this efficiency as central to PEEK’s practical value.
4.5 Qualitative analysis
Qualitative comparisons show that PEEK and the frame-wise Oracle generally identify the same globally salient regions, while PEEK produces smoother temporal relevance profiles and can preserve locally important content missed by uniform sampling.
- Qualitative comparison: PEEK and the frame-wise Oracle agree on globally salient regions but often differ in local frame-level selections.The comparison uses PEEK and SigLIP2 scores on ActivityNet Captions test segments.
- Temporal profiles: PEEK produces smoother temporal profiles than the frame-wise Oracle.This difference is visible in the qualitative score comparisons shown in Figure 3.
- Example: In one example, PEEK and the Oracle identify bagpipes, whereas the uniform center frame misses the instrument.Additional qualitative examples are provided in the supplementary material.
4.6 Ablations
Ablations on MSR-VTT with Qwen2.5-VL-3B show that stratified argmax selection and ListMLE distillation improve PEEK over alternative frame-selection and loss designs. The results emphasize temporal coverage and listwise ranking as important components.
- Ablation setup: The ablation evaluates fixed-size frame-set conversion and teacher-ranking distillation loss on MSR-VTT with Qwen2.5-VL-3B.The two design choices are inference-time frame selection and the training objective used to distill teacher rankings.
- Frame selection: Stratified argmax consistently improves over raw top-k across all metrics and budgets by preserving temporal coverage across equal video bins.Raw top-k selects the k highest-scoring frames globally, whereas stratified argmax selects one highest-scoring frame per temporal bin, avoiding near-duplicates.
- Distillation loss: ListMLE improves all metrics at k=1 and k=2 over MSE combined with pairwise ranking loss, with the largest CIDEr gain at k=1.This result motivates using a listwise objective for distilling the teacher’s frame relevance ranking.
5 Discussion and limitations
PEEK is most effective under tight frame budgets, but uniform sampling remains competitive when more frames are available. Its caption-derived Oracle supervision and short-caption evaluation may limit relevance beyond reference-aligned events and compact descriptions.
- Effectiveness under tight budgets: PEEK is the best query-free selector in all one-frame CIDEr settings and most two-frame settings across both benchmarks.These results support recovering part of the Oracle teacher’s caption-conditioned relevance signal from visual evidence alone.
- Effectiveness under tight budgets: Uniform sampling remains a strong baseline when several frames are forwarded, often achieving the best CIDEr on MSR-VTT at k=4.Short videos can be adequately covered by a few uniformly spaced frames.
- Effectiveness under tight budgets: As the frame budget increases, selecting the single most relevant frame becomes less valuable than temporal coverage and visual diversity.This trade-off is especially relevant for the relatively short ANC segments and MSR-VTT clips.
- Supervision limitations: Ground-truth-caption teacher supervision ties learned relevance to reference-caption alignment rather than all visually meaningful events.Frames supporting correct but non-reference captions may receive weak teacher scores, while reference-based metrics can penalize differing correct captions.
- Evaluation limitations: The short-caption evaluation does not establish performance for long-form video captioning involving multiple events, fine-grained temporal order, and additional details.Caption-aligned selection could overemphasize the dominant event and discard secondary events.
6 Conclusion
PEEK is a query-free frame selector that distills caption-conditioned relevance rankings into a lightweight temporal model for inference without target captions. Its advantages are strongest under tight frame budgets, while offering a favorable efficiency trade-off against adaptive baselines.
- Method and contribution: PEEK distills caption-conditioned relevance rankings from an Oracle teacher into a lightweight temporal model that selects frames without caption queries.The Oracle estimates caption-aware selection, while PEEK makes part of that signal available when the target caption is unavailable.
- Frame-budget results: 14 out of 16 CIDEr settings were won by PEEK on ActivityNet Captions, while MSR-VTT transfer was strongest at k=1 and k=2.At k=4 and k=8 on MSR-VTT, Uniform, Random, or MaxInfo occasionally performed slightly better.
- Efficiency and applications: PEEK is much faster than CSTA and MaxInfo while consistently outperforming them in the low-frame regime.This efficiency–accuracy trade-off makes PEEK practical for efficient video captioning and relevant to thumbnail or preview-frame selection.