PaperlayerView exact revision on arXiv

Source-linked AI summary

LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs

Jihwan Kim, Nikhil Parthasarathy, Danfeng Qin, Junhwa Hur, Deqing Sun, Bohyung Han, Ming-Hsuan Yang, Boqing Gong

arXiv:2605.17260v2cs.CV

TL;DR

Long-form Video LLMs face a visual-token scaling problem: post-hoc reduction lowers LLM cost but shifts latency to per-frame vision encoding. LiteFrame trains a compact student encoder with Compressed Token Distillation and Language Model Adaptation to predict compressed teacher representations. Compared with InternVL3-8B, it processes 8× more frames with 35% lower end-to-end latency and improves the efficiency–accuracy trade-off across video benchmarks.

  • Problem

    Long-form video scaling is limited by exploding visual-token context and expensive per-frame vision processing, while existing methods mainly reduce tokens after feature extraction.

  • Method

    LiteFrame uses Compressed Token Distillation to train a compact student encoder to predict information-dense, spatio-temporally compressed teacher representations, followed by Language Model Adaptation.

  • Results

    Compared with InternVL3-8B, LiteFrame processes 8× more frames with a 35% reduction in end-to-end latency and improves average video understanding accuracy across multiple benchmarks.

  • Takeaways & Limitations

    LiteFrame establishes a new latency–accuracy Pareto frontier by bypassing redundant full-resolution teacher computation through a compact encoder.

  • Takeaways & Limitations

    LMA uses a subset of existing video data, and LiteFrame is evaluated exclusively on videocentric benchmarks; performance on purely static image benchmarks remains unexplored.

Abstract

from arXiv · show

The fundamental challenge in scaling Video Large Language Models (Video LLMs) to long-form video lies in managing the explosion of visual-token context length. Existing strategies predominantly focus on "post-hoc" token reduction -- reducing visual tokens after feature extraction to alleviate the LLM's computational overhead. While these methods effectively reduce the number of visual tokens, we observe that the primary latency bottleneck then shifts from the LLM to the expensive per-frame processing of the vision encoder. To address this, we introduce LiteFrame, a strong, yet highly efficient video encoder backbone for Video LLMs. To train LiteFrame, we propose Compressed Token Distillation (CTD), a novel training framework that teaches a compact student vision encoder to directly predict information-dense, spatio-temporally compressed representations produced by a large teacher vision model, effectively bypassing redundant computation. When coupled with further Language Model Adaptation (LMA), this approach results in a new latency-accuracy Pareto frontier -- compared with InternVL3-8B, LiteFrame provides a 35% reduction in end-to-end latency while processing 8$\times$ more frames and improves average video understanding accuracy across multiple benchmarks. Our results demonstrate a new potential path to unlocking longer-form video understanding under fixed compute budgets.

1. Introduction

Long-form video understanding is limited by rising visual-token costs: post-hoc reduction shifts latency from the LLM to per-frame vision encoding. LiteFrame addresses both bottlenecks with a compact encoder trained by CTD and paired with LMA, improving the latency–accuracy trade-off.

  • Motivation: Video LLMs face prohibitively increasing computational costs as frame counts grow because spatio-temporal video data produces more visual tokens.The standard architecture includes an image encoder followed by an LLM, and the scaling problem becomes acute for long-form video.
  • Motivation: Post-hoc token reduction alleviates LLM computation but leaves per-frame feature extraction unchanged, making the vision encoder the new latency bottleneck.This bottleneck imposes a floor on achievable end-to-end inference efficiency as frame counts increase.
  • Approach: LiteFrame is a lightweight video encoder that reduces per-frame compute by directly predicting information-dense, spatio-temporally compressed teacher representations.Its student architecture also reduces spatio-temporal redundancies across frames.
  • Results: 35% lower end-to-end latency and 8× more frames are achieved by LiteFrame versus InternVL3-8B, while the encoder uses 87M parameters versus the teacher’s 304M.The result is reported with lightweight Language Model Adaptation, which adapts the new encoder and LLM together.
  • Results: LiteFrame and Weighted Average Pooling push the video-understanding efficiency Pareto frontier, whereas existing post-hoc methods neglect encoder latency.The comparison uses average accuracy across four video benchmarks and end-to-end latency including vision encoding and LLM prefilling.
  • Results: The paper reports a 1.53× end-to-end inference acceleration while processing 8× more frames and outperforming baselines across multiple video understanding tasks.This summarizes the reported contribution-level comparison with the InternVL3-8B teacher.

2. Related work

Related approaches reduce visual tokens after heavy frame-level encoding, but this remains inefficient for long videos because encoder latency becomes dominant. Other efficient encoders address parts of the bottleneck while exposing either dense tokens to the LLM or retaining the heavy encoder.

  • Post-hoc token reduction: Post-hoc token reduction applies spatial, spatio-temporal, or query-guided pruning after a frozen image encoder extracts frame-level features.This extract-and-reduce paradigm primarily targets the LLM’s quadratic self-attention cost.
  • Frame-count scaling: Figure 3 links increasing input frames to logarithmic accuracy growth on Video-MME, MLVU, and LongVideoBench, while conventional models remain capped at approximately 64 frames.The cap reflects both LLM context limits and dense per-frame tokenization.
  • Post-hoc token reduction: Post-hoc methods remain inefficient for hundreds or thousands of frames because the heavy encoder processes every frame before compression.Their latency bottleneck therefore shifts from the LLM to vision encoding.
  • Efficient vision encoders: Efficient vision encoders such as MobileNet-v5 and FastVLM target image encoding, while Video-Panda removes the backbone but exposes dense tokens to the downstream LLM.AutoGaze pre-filters tokens before ViT processing, but the related-work passage is truncated before its full limitation is stated.

3. Revisiting Post-Hoc Reduction

Revisiting post-hoc reduction motivates Weighted Average Pooling as a simple compression primitive and shows that aggressive compression can exchange redundant tokens for richer temporal context. Because post-hoc reduction leaves encoder cost unchanged, LiteFrame internalizes compression in a student encoder.

  • WAP motivation: Weighted Average Pooling is presented as a simple compression primitive that compares favorably with more complex token merging and pruning strategies.The paper uses WAP both for token reduction and to generate supervision targets for distillation.
  • WAP results: Under 16× compression, WAP achieves the highest average accuracy among the evaluated post-hoc token reduction strategies on three video benchmarks.The evaluation applies the methods to InternVL3-8B with 64 frames.
  • WAP mechanism: WAP partitions feature tensors into non-overlapping spatio-temporal blocks and produces compressed tokens by retaining high-activation features.The operation reduces token count by a compression factor r.
  • Frame-count bottleneck: Accuracy on Video-MME, MLVU, and LongVideoBench grows logarithmically with input frames, but conventional models are practically capped at approximately 64 frames.Dense per-frame tokenization contributes to the frame-count bottleneck.
  • Frame-count bottleneck: 16× WAP compression enables 16× more frames under a fixed visual-token budget and achieves the best results among the tested variants.The comparison supports trading redundant tokens for richer temporal context.
  • Scaling paradox: Post-hoc reduction does not reduce base encoder computation, so encoder latency explodes as frame counts scale and becomes the new bottleneck.This motivates internalizing token compression within a customized compact student network.

4. LiteFrame: Internalizing Spatio-Temporal Token Compression

LiteFrame resolves Video LLM scaling bottlenecks by internalizing spatio-temporal token compression in a lightweight encoder. CTD trains the student to predict compressed teacher representations, while LMA adapts its latent space to the LLM and extended temporal context.

  • Overview: LiteFrame addresses both the LLM’s quadratic visual-token cost and the vision encoder’s latency when processing many video frames.Unlike post-hoc methods, it compresses tokens within the video backbone.
  • Architecture: 87M parameters versus 304M for the teacher makes the student substantially more compact.The student uses a 12-layer, 768D ViT-Base, while the teacher uses a 24-layer, 1024D ViT-Large.
  • Architecture: Depth-Wise Temporal Convolutions achieve the lowest latency and FLOPs among compared temporal operators, with less than 1M parameter overhead.They are evaluated using 256 input frames and introduce spatio-temporal modeling alongside spatial attention.
  • Compressed Token Distillation: CTD trains the student to match WAP-compressed teacher features rather than redundant dense teacher outputs.The student minimizes MSE against compressed representations, transferring WAP’s attention-based weighting into static network parameters.
  • Language Model Adaptation: LMA fine-tunes the encoder and LLM with video-text pairs using LoRA to align the student’s latent space and support context up to 512 frames.This stage uses standard cross-entropy loss for video-conditioned text generation.
  • Results: 8× more frames and up to 35% lower end-to-end latency establish a superior latency-accuracy trade-off against the baseline.Table 3 compares LiteFrame and FastVID on InternVL3-8B under comparable total latency budgets.

5. Experiments

Experiments show that LiteFrame establishes a strong latency–accuracy trade-off across long-video, spatial-scaling, and efficient-encoder comparisons. Its compressed student architecture, CTD objective, temporal convolutions, and LMA jointly support this frontier.

  • Long-video scaling: LiteFrame reduces total inference latency by up to 35% while improving average accuracy from 65.3% to 65.7% on multiple benchmarks.Under a restricted latency budget, the accuracy gap widens to 2.1 percentage points: 61.1% versus 59.0%.
  • Long-video scaling: 8× more frames can be processed within a fixed compute budget by trading spatio-temporal redundancy for richer temporal context.The comparison uses InternVL3-8B as the primary baseline.
  • Post-hoc compression: CTD surpasses post-hoc compression methods in both efficiency and accuracy because compression is learned within the encoder rather than applied after expensive feature extraction.The post-hoc baselines remain bottlenecked by computation incurred before compression.
  • Spatial resolution scaling: 54.1 state-of-the-art accuracy on HLVid is achieved at 2688px with 48 frames, surpassing AutoGaze’s 52.6 at 3584px and 1024 frames.LiteFrame’s token efficiency preserves a better balance between spatial and temporal resolution as input resolution increases.
  • Efficient vision encoders: LiteFrame is 1.2× faster than VideoPanda and 3.3× faster than FastVLM while also reducing downstream computational cost through compact token outputs.The comparison fine-tunes InternVL3-8B with LoRA and uses frozen visual encoders.
  • Ablation studies: The full CTD + LMA configuration achieves 63.4 average accuracy at 87.4 latency, outperforming alternatives in the ablation study.Depth-wise temporal convolutions outperform full spatio-temporal attention, while RTD substantially lags CTD.

6. Conclusion

The paper identifies the vision encoder as the limiting latency bottleneck after post-hoc token reduction and introduces LiteFrame to internalize spatio-temporal compression. Across long-video benchmarks, it reports faster inference, better accuracy, and 8× more frames processed.

  • Conclusion: Post-hoc token reduction lowers LLM cost but leaves the vision encoder as a prohibitive bottleneck at high frame counts.LiteFrame addresses this by using a compact encoder trained with CTD and LMA to predict dense pooled teacher tokens.
  • Conclusion: LiteFrame achieves 35% faster end-to-end inference and better accuracy while processing 8× more frames across multiple long-video benchmarks.The approach trades spatio-temporal redundancy for richer temporal context.

A. Implementation details

The implementation uses InternVL3-8B’s image encoder as the teacher and a smaller ViT-Base student, with CTD and LMA trained and evaluated under specified hardware, sampling, and benchmark protocols.

  • Training: CTD training uses AdamW, cosine scheduling with linear warmup, a global batch size of 512, and eight NVIDIA H100 GPUs.The maximum learning rate is 4e-5, with outlier and gradient clipping used for stabilization.
  • Training: LMA adapts the LLM with LoRA using rank r=4, α=8, dropout 0.05, and frame counts sampled from {128, 256, 512}.The effective batch size is 128 and the learning rate is 4e-5 with cosine scheduling.
  • Evaluation: Evaluation covers Video-MME, MLVU, LongVideoBench, HLVid, MVBench, and TVBench, with latency measured end-to-end over visual encoding and LLM prefilling.Latency is reported as the median over 100 iterations after 40 warmup iterations on one NVIDIA A100-80GB GPU.
  • Evaluation: LiteFrame uses dense clip sampling with four-frame clips at a minimum of 1 FPS, while baseline evaluation uniformly samples frames and resizes inputs to 448px.The two sampling protocols are specified separately for LiteFrame and InternVL3-8B/post-hoc comparisons.

B.1. Short video benchmarks

LiteFrame extends its efficiency gains to short-video benchmarks, reducing end-to-end latency while maintaining baseline accuracy on both MVBench and TVBench.

  • Short video benchmarks: 28% lower end-to-end latency on MVBench is achieved while maintaining baseline accuracy.The result is reported for the short-video evaluation setup.
  • Short video benchmarks: 63% lower end-to-end latency on TVBench is achieved while maintaining baseline accuracy.The reduction is larger than the reported MVBench reduction.

B.2. Long video benchmarks

LiteFrame maintains or improves long-video understanding while extending frame counts and reducing latency, especially on LVBench and in low-latency MMBench-Video settings.

  • 43.9 vs. 43.5: LiteFrame scores higher on LVBench with 512 frames than the 64-frame baseline.It also operates 38% faster while using the extended temporal context.
  • 38% faster: LiteFrame achieves the superior 43.9 LVBench score at a 512-frame input budget.
  • LiteFrame improves efficiency in the low-latency region of MMBench-Video.The reported regime covers 16–128 input frames.
  • 43.9 vs. 43.5: LiteFrame reaches the best reported LVBench score compared with the 43.5 reference.

C.1. Reconstructive training objective

The section contrasts reconstructive token distillation with CTD and evaluates compression strategies, showing that CTD with spatio-temporal compression offers the strongest supported trade-offs.

  • Reconstructive Token Distillation: RTD trains the student as an encoder and auxiliary transformer blocks as a decoder to reconstruct the teacher’s dense feature map.Its objective uses the student’s compressed latent representation to recover the teacher output.
  • Reconstructive Token Distillation: RTD seeks a compression manifold that preserves general visual information and can discover spatio-temporal dependencies.
  • Distillation ablation: CTD consistently outperforms RTD, and CTD without LMA already surpasses RTD with LMA.The comparison supports aligning the student with the WAP primitive rather than using generic reconstruction.
  • Distillation ablation: 65.3% at 256 frames: CTD combined with LMA achieves the highest accuracy across all frame budgets.CTD distills teacher features, while LMA aligns the pretrained LLM with the compressed student representations.
  • Compression strategies: 16 tokens per frame: LiteFrame enables 8× more frames with 272.6ms latency versus 393.3ms for CTD at 256 input frames.The compressed token volume reduces the visual-token load associated with prefilling.
  • Spatio-temporal vs. spatial-only compression: 62.8% vs. 60.5%: spatio-temporal compression beats spatial-only compression in average accuracy at 128 frames.Video-MME fine-grained spatial understanding is 61.9% versus 57.0%, while distributing compression preserves spatial details and aggregates temporal redundancy.

D.2. Evaluation setups for AutoGaze

The AutoGaze evaluation uses author-recommended settings, matched input resolution, preprocessing-excluded neural execution latency, and median latency on Video-MME.

  • Evaluation configuration: AutoGaze is evaluated with the authors’ optimal hyperparameters, 448px input resolution, and thumbnail frames set to num_video_frames//16.
  • Latency measurement: Latency excludes AutoGaze video preprocessing and uses median latency on Video-MME because latency varies across videos.
  • Baseline definitions: ImageViT encodes videos frame-wise, whereas VideoViT processes all visual tokens from each 16-frame clip with full spatio-temporal attention.

E. Limitations

The paper identifies limitations in LMA data coverage, benchmark scope, and the stability of scaling CTD to smaller student models.

  • Data coverage: LMA uses only a subset of existing video data, leaving higher-quality extreme long-form datasets as a potential avenue for improvement.
  • Evaluation scope: LiteFrame is evaluated exclusively on videocentric benchmarks, so performance on purely static image benchmarks remains unexplored.
  • Model scaling: 304M to 87M parameters: scaling CTD to even smaller students was constrained by training instabilities such as loss explosions.
Loading 2605.17260v2…