PaperlayerView exact revision on arXiv

Source-linked AI summary

Elephant in the Fridge: Constant-Memory Frame Packing for Long Video Understanding

Shuo Gao, Chenhao Zheng, Zhongzheng Ren

arXiv:2610.16118v1cs.CVcs.LGcs.MM

TL;DR

Long-video VLMs face growing visual-token sequences that increase memory use and inference cost, while existing compression methods still scale with video length. FramePack-U ranks frames for prompt relevance and visual diversity, then packs them at heterogeneous resolutions under a fixed token budget. It consistently outperforms full-resolution and prior compression methods across the reported benchmarks.

  • Problem

    Frame-wise video encoding makes visual-token counts grow with video length, and existing token reduction or keyframe selection still leaves memory and latency scaling with duration.

    Source: §1 · p. 2

  • Method

    FramePack-U ranks frames by instruction relevance and visual diversity, then packs them at hierarchical spatial resolutions under a fixed token budget.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    FramePack-U consistently outperforms full-resolution and prior token-reduction and keyframe-selection methods across benchmarks, with gains of 12.9%, 10.6%, and 9.7% on LongVideoBench, MLVU, and Video-MME (long) for Qwen3.5-35B-A3B at a 128-frame budget.

    Source: §1 · p. 2

  • Takeaways & Limitations

    FramePack-U offers a practical step toward long-video understanding as a plug-and-play module for off-the-shelf VLMs.

    Source: §7 · p. 14

    FramePack-U is less plug-and-play for grid-based or tightly coupled video encoders, which may require changes to the visual encoder or preprocessing pipeline.

    Source: §6 · p. 13

Abstract

from arXiv · show

Hour-long videos, streaming feeds, and movie-length content are pushing vision-language models (VLMs) past their breaking point -- the standard recipe of encoding each frame into visual tokens yields sequences that grow linearly with video length, causing prohibitive memory consumption and inference latency. Existing approaches, such as token reduction and keyframe selection, shorten the token sequence but leave the O(n) scaling intact, failing to resolve the fundamental bottleneck. We propose FramePack-U, a strikingly simple, training-free frame packing framework that compresses arbitrarily long videos into a fixed size visual representation at constant memory cost. FramePack-U requires no fine-tuning or architectural modification -- it plugs directly into off-the-shelf VLMs at inference time. Inspired by FramePack from the generation literature, we propose a content-aware compression strategy for visual understanding under a fixed token budget. Specifically, FramePack-U (1) scores each frame by its semantic similarity to the text prompt and ranks frames in a diversity-driven manner, and (2) packs the selected frames at varying resolutions into a fixed-length context window. This concentrates the memory budget on frames that are both instruction-relevant and visually diverse. While FramePack-U can in theory pack arbitrarily many frames into the fixed budget, we identify an empirical instantiation that generalizes robustly across settings. Extensive experiments on LongVideoBench, MLVU, and Video-MME show that FramePack-U consistently outperforms full-resolution baselines and prior token reduction and keyframe selection methods across diverse model families, parameter scales, and frame budgets. Notably, on Qwen3.5-35B-A3B with a 128-frame budget, FramePack-U yields gains of +12.9\%, +10.6\%, and +9.7\% over the official model on the three benchmarks, respectively.

1 Introduction

Long-video VLMs face visual-token growth that raises memory and inference costs, while existing compression still scales with video length and can erase transient actions. FramePack-U uses relevance- and diversity-aware frame ranking and heterogeneous-resolution packing to keep memory fixed, and improves benchmark performance over full-resolution and prior compression baselines.

1 Introduction

  • Existing token reduction and keyframe selection leave retained-token counts scaling as O(n) or more and may erase transient actions through hard keep-or-discard decisions.The introduction identifies lost temporal continuity and transient actions as concerns for long-horizon reasoning.Source: §1 · p. 2
  • FramePack-U creates a fixed-size representation with constant O(1) memory, without fine-tuning or architectural changes to off-the-shelf VLMs.It adapts heterogeneous-resolution packing from FramePack for understanding, rather than generation.Source: §1 · p. 2
  • It ranks frames by prompt similarity and visual distinctness, then assigns finer resolutions to higher-ranked frames and progressively downsamples lower-ranked ones.The approach concentrates the fixed budget on relevant, diverse frames while preserving full temporal coverage.Source: §1 · p. 2
  • FramePack-U consistently outperforms full-resolution baselines and prior token-reduction and keyframe-selection methods across benchmarks, model families, scales, and frame budgets.On Qwen3.5-35B-A3B with a 128-frame budget, it gains 12.9% on LongVideoBench, 10.6% on MLVU, and 9.7% on Video-MME (long).Source: §1 · p. 2
  • 12.9%, 10.6%, and 9.7% are the reported gains on LongVideoBench, MLVU, and Video-MME (long), respectively, for Qwen3.5-35B-A3B with a 128-frame budget.The results also report consistent gains on Qwen3.6-27B: 6.4%, 8.0%, and 2.7% on those benchmarks, respectively.Source: §1 · p. 2

2 Related Work

Related work covers frame-wise VLM encoders, methods for compacting long-video representations, and heterogeneous frame packing for generation. FramePack-U adapts the packing principle to inference-time video understanding.

Vision-Language Models

  • Video VLMs commonly apply image encoders frame by frame, while some approaches use pretrained video encoders on frame pairs or short clips.Scaling video encoders to longer temporal windows still trails per-frame image encoding with strong backbones such as SigLIP 2.Source: §2 · p. 3

Long Video VLM

  • Long-video compression includes visual-token pruning or merging, keyframe selection, and architectural approaches such as memory banks and temporal aggregation.Despite their differences, these approaches produce token sequences that still scale with video duration.Source: §2 · p. 3

Packing for Generation

  • FramePack allocates heterogeneous token budgets across frames for video generation, and FramePack-U adapts that packing principle to inference-time VLM understanding.Related approaches also use history compression for avatar reconstruction and interactive world generation.Source: §2 · p. 3

3 FramePack-U

FramePack-U constructs a visual sequence within a fixed token budget by sampling frames for relevance and diversity, then assigning them hierarchical spatial resolutions. This allows broader temporal coverage while concentrating finer resolution on higher-priority frames.

3 FramePack-U

  • FramePack-U first samples candidate frames using instruction relevance and visual diversity, then assigns each frame a spatial downsampling factor under a fixed token budget.The packed sequence is passed to the VLM as visual tokens.Source: §3 · p. 3
  • The candidate count N can in principle reach T, but under a fixed budget, increasing N trades per-frame resolution for temporal coverage.The best N depends on model capacity and pre-training data because performance may degrade when resolution falls below the pre-training range.Source: §3 · p. 3

3.1 Relevance- and diversity-aware frame sampling

  • Greedy sampling starts with the most prompt-relevant frame and selects subsequent frames for visual distinctness, balancing relevance with non-redundant coverage.The procedure continues until N frames have been selected.Source: §3.1 · pp. 4–5

3.2 Hierarchical resolution packing

  • Hierarchical resolution packing assigns finer spatial resolutions to informative frames and progressively compresses the rest to maximize temporal coverage within the budget.The packed frames receive different spatial downsampling factors.Source: §3.2 · p. 5
  • A d1f16d2f448 schedule allocates 16 full-resolution frames and 448 frames at 2× downsampling, costing 128 full-resolution frame equivalents.A query-agnostic binary-tree midpoint ordering is also considered as a baseline for uniform temporal coverage.Source: §3.2 · p. 5
  • Different resolution combinations can use the same memory while retaining different numbers of frames; the ranking gives finer levels to frames selected earlier.The priority ranking concentrates the high-resolution quota on frames that are both instruction-relevant and visually diverse.Source: Figure 2 · p. 4, §3.2 · p. 5

4 Experiments

Experiments compare FramePack-U with training-free compression methods under equal visual memory budgets, then examine resolution allocation, selection components, model scale, and frame coverage.

4.1 Results

4.2 Ablations

  • All tested mixed-resolution schemes beat vanilla, though the best scheme varies by benchmark; gains persist across 608–1664 packed frames under a 512-frame budget.Source: §4.2 · p. 7, Table 2 · p. 8
  • Relevance-only can spend high-resolution slots on similar frames; combining relevance and diversity selects more complementary frames and yields more reliable performance at fixed memory.Source: Table 4 · p. 8, §4.2 · p. 8
  • Under 512-frame memory, LongVideoBench accuracy rises from 74.0 to 74.9 and from 73.1 to 74.6 with prompt relevance for two schemes.Source: §4.2 · p. 8, Table 4 · p. 8

4.3 Qualitative results

  • Relevance-only sampling tends to select redundant frames, uniform sampling may miss question evidence, and the combined strategy captures key evidence while covering the video.Source: Figure 3 · p. 9, §4.3 · pp. 9–10

4.2 Ablations

  • FramePack-U’s benefit grows with model scale; smaller 4B and 9B models show less stable gains and can degrade as packed frames increase.Source: Table 3 · p. 8, §4.2 · p. 9

4.3 Qualitative results

  • At the same token-memory footprint, FramePack-U represents 848 frames versus uniform sampling’s 128, with denser feature coverage and resolution tiers for prioritizing frames.Source: Figure 4 · p. 10, §4.3 · p. 11

5 Runtime analysis

FramePack-U keeps memory use constant as video duration grows and reduces VLM inference latency. Its end-to-end latency can nevertheless rise on extremely long videos when decoding many more candidate frames adds I/O overhead.

Memory overhead

  • FramePack-U maintains a constant memory upper bound as frame counts grow, while vanilla-model input-token memory increases substantially.The CLIP-based scoring process adds constant, relatively trivial memory overhead.Source: §5 · p. 11, Table 5 · p. 12

Inference latency

  • FramePack-U reduces VLM inference and overall latency versus vanilla models, but Qwen3.6-27B has higher overall latency on extremely long videos because of increased I/O.The comparison restricts vanilla Qwen3.6-27B to 224 frames, while FramePack-U decodes up to 3,600 frames for a 3,600-second video; under equal 1-fps sampling, the authors say FramePack-U would have lower overall latency and vanilla could run out of memory or suffer catastrophic delays.Source: §5 · pp. 11–13, Table 6 · p. 13
  • The CLIP-based preprocessing latency is relatively negligible compared with VLM inference latency.Latency evaluations used Qwen3.5-4B, Qwen3.5-9B, and Qwen3.6-27B on one NVIDIA A800 GPU, with candidate frames sampled at 1 fps.Source: §5 · p. 11

6 Limitations

FramePack-U’s benefits depend on the base VLM’s reasoning capacity and on the video encoder’s input design. Its relevance scoring and resolution-allocation choices also have limits that can affect which frames receive fidelity.

Model capacity

  • Smaller VLMs may gain inconsistently from larger frame budgets, as added visual observations can distract models with insufficient visual reasoning capacity.On smaller Qwen-series models, more packed frames sometimes improve LongVideoBench but slightly degrade MLVU or Video-MME.Source: §6 · p. 13

Video encoder compatibility

  • FramePack-U is less plug-and-play for grid-based or jointly processed video encoders because per-frame resolution changes can break their input format or require non-trivial modifications.The authors identify Eagle2.5-style architectures as an example.Source: §6 · p. 13

Relevance and diversity scoring

  • CLIP similarity may under-prioritize frames containing fine-grained temporal events, subtle interactions, or reasoning-heavy cues that are difficult to recognize from individual frames.The diversity term based on visual feature similarity may also fail to capture semantic complementarity.Source: §6 · pp. 13–14

Resolution allocation

  • The optimal resolution-allocation scheme can vary across models, benchmarks, and video domains, leaving a scheme-selection hyperparameter for users to choose.The authors suggest automatic selection based on video length, query type, and model capacity as a future improvement.Source: §6 · p. 14

7 Conclusion

FramePack-U packs long-video frames at heterogeneous resolutions under a fixed token budget, with constant O(1) memory complexity. It combines instruction relevance and visual diversity, and the authors report consistent benchmark gains over full-resolution and prior compression baselines.

Conclusion

  • FramePack-U combines instruction relevance and visual diversity to rank frames, then allocates heterogeneous resolutions under a fixed token budget while preserving full temporal coverage.The method is training-free and has constant O(1) memory complexity.Source: §7 · p. 14
  • Across LongVideoBench, MLVU, and Video-MME, FramePack-U consistently outperforms full-resolution baselines and prior token-reduction and keyframe-selection methods.The conclusion also reports that it frequently matches or exceeds substantially larger proprietary VLMs.Source: §7 · p. 14
  • As a plug-and-play module for off-the-shelf VLMs, FramePack-U offers a practical step toward scalable long and streaming video understanding.Source: §7 · p. 14

A Alternative strategies

The alternative strategies use temporal structure or query similarity to rank frames, but each has a distinct constraint. Binary-tree midpoint ordering spreads selected frames across time, while query-only ranking can favor visually redundant frames.

Binary-tree midpoint ordering

  • Chronological ranking concentrates full-resolution priority on early frames, which may discard later temporal context because early frames are not necessarily more informative.Source: A Alternative strategies · p. 18
  • Binary-tree midpoint ordering ranks temporal midpoints breadth-first, distributing any smaller anchor-frame budget across the video’s full temporal extent.This ordering allocates higher fidelity to temporally representative moments.Source: A Alternative strategies · p. 18

CLIP-guided sampling

  • CLIP-guided ranking selects and assigns higher resolution to frames with greater text-frame similarity, but query-only ranking may select visually similar frames and miss complementary evidence.The default strategy adds a diversity-aware greedy selection term to encourage instruction-relevant, visually non-redundant frames.Source: A Alternative strategies · pp. 18–19

B Additional analysis on FramePack-U

FramePack-U’s ablations show that both prompt relevance and visual diversity contribute to packing performance, while results vary with model size, memory budget, and benchmark. Larger models generally benefit more, and no single resolution allocation is optimal across all benchmarks.

Resolution allocation

  • At 128 frames, d1f16d2f320d4f512 performs best on LongVideoBench and MLVU, while d1f96d2f128 performs best on Video-MME.At 256 frames, several schemes are comparable on LongVideoBench and MLVU, while d1f250d4f96 performs best on Video-MME; the preferred balance varies by benchmark.Source: B Additional analysis on FramePack-U · pp. 19–20

Frame diversity

  • Removing diversity lowers Video-MME from 76.3 to 68.9 with d1f16d2f448 and also degrades other tested schemes at both budgets.The ablation ranks frames only by query-frame similarity; the authors report that diversity helps prevent redundant frames from using the limited high-resolution budget.Source: Table 8 · p. 20, B Additional analysis on FramePack-U · p. 20

Prompt relevance

Model size

  • Qwen3.6-27B gains +5.6, +7.2, and +3.8 on LongVideoBench, MLVU, and Video-MME at 128 frames; smaller models gain more moderately.At 512 frames, Qwen3.6-27B improves across benchmarks, while the smaller models show slight degradation on MLVU or Video-MME.Source: Table 9 · p. 21, B Additional analysis on FramePack-U · p. 21
  • At larger budgets, smaller models show weaker or less stable improvements, which the authors attribute as likely reflecting limits in reasoning and visual integration.The authors report that Qwen3.6-27B continues to gain across benchmarks at 512 frames, while Qwen3.5-4B and Qwen3.5-9B improve on LongVideoBench but slightly degrade on MLVU or Video-MME.Source: B Additional analysis on FramePack-U · p. 21

C Fine-grained results

Fine-grained results show FramePack-U improving performance across most duration splits, with particularly clear gains on long-video subsets. The reported long-duration gains occur under both 128- and 256-frame budgets and across two model settings.

Duration-split results

  • FramePack-U improves performance across most fine-grained duration splits, with gains especially pronounced on longer videos.The results cover LongVideoBench’s 15s, 60s, 600s, and 3600s subsets and Video-MME’s short, medium, and long subsets.Source: C Fine-grained results · p. 21, Table 10 · p. 22

Long-duration results

  • On Qwen3.5-35B-A3B at 128 frames, LongVideoBench 600s and 3600s accuracy rises from 55.3 and 50.2 to 73.1 and 64.7, while Video-MME long rises from 54.1 to 66.9.At 512 frames, LongVideoBench 3600s improves from 50.2 to 66.1 and Video-MME long from 56.3 to 66.4.Source: Table 10 · p. 22, C Fine-grained results · p. 22
  • On Qwen3.6-27B at 128 frames, LongVideoBench 3600s accuracy rises from 58.0 to 69.1 and Video-MME long from 61.4 to 65.7.At 256 frames, LongVideoBench 3600s improves from 61.5 to 68.6 and Video-MME long from 65.6 to 68.3.Source: Table 10 · p. 22, C Fine-grained results · p. 22

D More comparisons with similar methods

FramePack-U differs from compared token-compression and keyframe-selection methods in how it ranks frames and assigns resolutions. In controlled comparisons, the authors report superior performance for FramePack-U, while other methods show weaker robustness or degradation.

Method differences

  • Unlike SlowFast-LLaVA’s query-agnostic, temporally predetermined resolution assignment, FramePack-U assigns resolutions using content relevance and visual diversity.SlowFast-LLaVA uses a low-frame-rate, high-resolution Slow pathway and a high-frame-rate, spatially pooled Fast pathway.Source: D More comparisons with similar methods · p. 22

Empirical comparison

  • In the controlled comparison, FramePack-U outperforms Q-Frame and LongVU, while the alternatives show weaker robustness or degradation.The comparison uses the same CLIP-ViT-B/32 feature extractor, Qwen3.5-4B backbone, and d1f96d2f128 resolution scheme.Source: Table 11 · p. 23, D More comparisons with similar methods · p. 23

Method differences

  • Unlike Q-Frame’s probabilistic Gumbel-Max sampling and static thresholds, FramePack-U uses deterministic hierarchical optimization under a strict constant-memory constraint.The passage states that this constraint guarantees O(1) memory complexity regardless of video length.Source: D More comparisons with similar methods · p. 23
  • Unlike LongVU’s fine-tuning and query-agnostic frame discarding, FramePack-U is training-free and ranks frames using semantic text relevance and visual diversity together.The passage describes LongVU as discarding frames based on DINOv2 visual similarity before introducing text queries.Source: D More comparisons with similar methods · p. 23

Related papers

Loading 2610.16118v1…