PaperlayerView exact revision on arXiv

Source-linked AI summary

Frames2LoRA: Parametric Video Internalization for Vision-Language Models

Manan Suri, Sarvesh Baskar, Dinesh Manocha

arXiv:2606.04351v2cs.CVcs.CL

TL;DR

Video VLMs repeatedly pay a large visual-token and context cost, motivating a way to internalize video before queries. FRAMES2LORA uses a Perceiver hypernetwork to generate video-specific LoRA adapters from frozen VLM representations in one pass, enabling text-only querying. It matches direct video-in-context inference across nearly all reported captioning and QA evaluations while reducing query-time token and latency costs.

  • Problem

    Video VLM inference repeatedly incurs large visual-token, memory, and latency costs as frames and queries increase, with context capacity becoming a bottleneck.

  • Method

    FRAMES2LORA uses a trainable Perceiver hypernetwork to convert frozen VLM layer-wise video representations into a video-specific LoRA adapter for text-only querying.

  • Results

    FRAMES2LORA is statistically non-inferior and equivalent to direct video-in-context inference across all five captioning benchmarks and seven of eight video-QA benchmark-scale pairings.

  • Takeaways & Limitations

    Video context can be internalized into generated adapter weights, supporting stable longer-video processing, substantially lower query-time token and latency loads, and rank-space chunk composition.

  • Takeaways & Limitations

    The current implementation trains separate hypernetworks for the 500M and 2.2B SmolVLM2 backbones and has not been extended to additional VLM families or shared scale-transferable hypernetworks.

Abstract

from arXiv · show

Processing video in vision-language models is expensive: each frame occupies hundreds of tokens, and inference cost scales with every frame and every repeated query. We introduce Frames2LoRA, a method for parametric video internalization. A perceiver hypernetwork reads the intermediate representations produced layer-by-layer as a frozen VLM encodes a video, and generates a Low-Rank Adaptation (LoRA) adapter in a single forward pass. Unlike standard LoRA fine-tuning, which requires iterative gradient updates, Frames2LoRA predicts these weights directly from the video. Trained for SmolVLM2 500M and 2.2B on video summarization and captioning, Frames2LoRA enables the same frozen VLM to answer queries from the adapter alone, with zero visual tokens in its context at query time. Frames2LoRA is statistically non-inferior and equivalent to direct video-in-context inference across all five captioning benchmarks at both model scales, and across seven of eight video question answering benchmark-scale pairings. Although trained only on 12 frames at 384px, it remains stable up to 1,024 frames and 1024px, where direct video-in-context inference often degenerates. Across this sweep, it reduces answer-time visual-token load by up to 1,500x and query TTFT by 6-80x, while preserving video-faithful outputs. We also find that independently generated adapters for non-overlapping video segments can compose in rank space, suggesting a path toward chunked long-video internalization.

1 Introduction

FRAMES2LORA addresses video VLMs’ context bottleneck by internalizing videos into generated LoRA adapters, eliminating visual tokens during querying. It matches direct video-in-context inference across broad captioning and QA evaluations while improving efficiency and supporting longer videos.

  • Motivation: Each video frame contributes hundreds of visual tokens, so memory and latency grow with every frame and repeated query.Beyond a capacity threshold, outputs can become incoherent or repetitive and unrelated to the video.
  • Approach: FRAMES2LORA converts a video into a LoRA adapter in one forward pass, allowing a frozen VLM to answer queries without visual tokens in context.Only the hypernetwork is optimized using cached teacher-generated captions and summaries as supervision.
  • Results: Up to 1,500× fewer answer-time visual tokens and 6–80× lower query TTFT are achieved while preserving video-faithful outputs.The method remains stable up to 1,024 frames and 1024px despite training only on 12 frames at 384px.
  • Extensions: Adapters generated independently for non-overlapping video segments can compose in rank space, suggesting chunked long-video internalization.This is presented as an observed compositionality result rather than a fully developed long-video method.

2 Related Work

Existing efficient video-understanding methods reduce token count, expand context, or maintain compact memories, but still retain visual information at query time. FRAMES2LORA instead stores video-specific information in adapter weights for later text-only querying.

  • Token and context management: Frame subsampling lowers token usage by discarding frames, sacrificing temporal coverage.
  • Token and context management: Visual-token compression prunes or merges spatial tokens, while long-context architectures enlarge the usable context window.
  • Memory-based methods: Streaming methods process video incrementally and maintain compact memory across time.
  • Parametric internalization: These approaches still condition the language model on visual tokens at query time, whereas FRAMES2LORA converts the video into adapter weights once.
  • Parametric internalization: FRAMES2LORA extends parameter-efficient context storage from methods such as gist tokens and Doc-to-LoRA to video-specific adapters.

3 FRAMES2LORA

FRAMES2LORA uses frozen VLM representations and a trainable Perceiver hypernetwork to generate video-specific LoRA weights in one pass. The resulting adapter is attached to the frozen answer model, which receives only downstream text prompts.

  • Architecture: A frozen VLM encodes the video into layer-wise hidden states, and a trainable Perceiver hypernetwork maps them into LoRA weights.
  • Architecture: The answer model receives the downstream text prompt and generated adapter, but not video tokens.
  • Video encoding: Keeping the layer dimension lets the hypernetwork generate layer-indexed adapters instead of one pooled video representation.
  • Hypernetwork: A Perceiver resampler uses learned latent queries and module- and rank-specific output queries to produce fixed-size adapter representations.
  • Dynamic LoRA injection: The generated adapter uses standard LoRA factors for selected frozen linear modules, with learned multipliers and zero-initialized B scaling.
  • Dynamic LoRA injection: Each video receives its own generated adapter, conditioning LoRA weights on the input video rather than sharing them across videos.

4 Experimental Setup

The experiments evaluate FRAMES2LORA on two frozen SmolVLM2 scales using controlled video captioning and video-QA benchmarks. Quality is measured with token-level F1 and an LLM judge, with paired bootstrap confidence intervals and predefined statistical margins.

  • Models and training: Two SmolVLM2 scales, 500M and 2.2B, are evaluated with frozen video encoder and answer-model backbones.Only the FRAMES2LORA hypernetwork is trained.
  • Models and training: Training uses 12 uniformly sampled frames at 384px longest-edge resolution from video spans derived from FineVideo.Generated LoRA adapters target text-decoder MLP down_proj modules with rank R = 16.
  • Benchmarks: Captioning covers ActivityNet Captions, PLM-RDCap, PLM-RCap, VDC, and CaReBench, while video QA covers NExT-QA, ActivityNet-QA, PLM-SGQA, and VidCapBench.
  • Evaluation protocol: Benchmark comparisons use matched videos, prompts, references, frame sampling, and decoding configurations for the direct baseline and FRAMES2LORA.Sample counts are fixed for scalable LLM-judge evaluation.
  • Metrics and statistics: Quality is measured with token-level F1 and a rescaled 1–5 LLM-judge score, whose human-study metric fidelity has Spearman ρ = 0.823.
  • Metrics and statistics: Paired bootstrap resampling supplies 95% confidence intervals, with non-inferiority and equivalence margins of 0.05 for token-F1 and 0.15 for rescaled judge score.

5 Results

FRAMES2LORA matches direct video-in-context inference across captioning and transfers zero-shot to most video QA settings, while improving efficiency and scaling robustness. Its results also support fine-grained behavior analysis and adapter composition for chunked videos.

  • Captioning: FRAMES2LORA passes non-inferiority and equivalence on all 10 captioning benchmark–scale combinations under both LLM Judge and Token-F1 evaluation.
  • Captioning: At 500M, captioning recovery spans 79–96%, while scaling to 2.2B raises the floor to 85% and enables PLM-RCap to surpass the base.
  • Captioning: Token-F1 mean paired deltas are −0.001 at 500M and 0.000 at 2.2B, with FRAMES2LORA exceeding the base on 3 of 5 and 2 of 5 benchmarks, respectively.
  • Fine-Grained Captioning: VDC camera captions are the main 500M weakness at 42.3% recovery, improving to 82.0% at 2.2B, while short and background captions exceed the base.
  • Fine-Grained Captioning: CaReBench temporal captioning is strongest at both scales, while holistic and spatial recovery improves at 2.2B and event recovery declines as the base improves.
  • Video Question Answering: Video QA is zero-shot transfer: FRAMES2LORA passes non-inferiority and equivalence on 7 of 8 benchmark–scale combinations, with NExT-QA outperforming the base at both scales.
  • Frame and Resolution Generalization: Despite training at 12 frames and 384px, FRAMES2LORA remains stable across frame-count and resolution sweeps, while direct inference can become repetitive or gibberish at large visual-token loads.
  • Inference Efficiency: FRAMES2LORA reduces query TTFT by geometric means of 6.7× at 500M and 20.1× at 2.2B, reaching maximum speedups of 17.2× and 79.1×.

6 Conclusion

FRAMES2LORA internalizes video context into generated LoRA adapters, enabling text-only querying after one forward pass. It matches direct video-in-context inference broadly, remains stable at long inputs, and supports rank-space composition.

  • FRAMES2LORA converts video into a LoRA adapter in a single forward pass, allowing a frozen VLM to answer queries without visual tokens.
  • It is statistically non-inferior and equivalent to direct video-in-context inference across five captioning benchmarks and seven of eight video-QA benchmark-scale pairings.
  • At 1,024 frames, FRAMES2LORA remains stable while direct inference degenerates, with 6–76× lower query latency and up to 1,500× fewer answer-time tokens.
  • Independently generated adapters can compose in rank space, supporting long-video internalization without dedicated training.

7 Limitations

The current method is bounded by its evaluated model families, supervision, detail preservation, and preliminary chunk-composition design. These limitations motivate broader models, mixed supervision, richer objectives, and structured composition.

  • The implementation trains a separate hypernetwork for each target VLM scale and is evaluated only on 500M and 2.2B SmolVLM2 backbones.
  • Captioning and summarization supervision makes video question answering zero-shot, and descriptive answers can lower lexical-overlap scores despite semantic appropriateness.
  • Compact adapters may emphasize high-level scene and event information over fine-grained camera, spatial, or object-level distinctions.
  • The chunk-composition experiment is an initial two-chunk test and does not explicitly model temporal order.

8 Ethics Statement

The paper reports no use of personally identifiable information and states that all datasets were used according to their licenses. The appendix also documents evaluation prompts and task-specific judging procedures.

  • The research does not use personally identifiable information, and all datasets are used in accordance with their respective licenses.
  • An LLM judge evaluates reference-based quality and output preservation using text-only inputs for the main reported scores.
  • Captioning judgments assess semantic coverage, while QA judgments extract and compare answers while allowing verbose responses and paraphrases.
  • The appendix provides evaluation prompts and templates for video description and question-answering tasks.

C.2 Analysis

Rank directions in generated adapters are redundant but not exchangeable: selected high-scoring slices preserve performance, while norm-based importance is useful but heuristic. Diagnostic procedures further examine adapter effects on the frozen answer model.

  • Rank-direction ablation: Top-8 rank slices achieve Token-F1 0.1264, close to 0.1262 for the full rank-16 adapter; top-4 reaches 0.1196.The top-4 adapter recovers 90.6% of the improvement over the zero-adapter baseline.
  • Rank-direction ablation: At k = 1, Top-k reaches Token-F1 0.0894 versus 0.0556 for Bottom-k and 0.0561 for the zero-adapter baseline.Random-k generally falls between Top-k and Bottom-k at matched budgets.
  • Rank-direction ablation: Removing the highest-norm slice raises the point estimate from Token-F1 0.1262 to 0.1317, but overlapping confidence intervals make this result suggestive rather than conclusive.
  • Rank-direction ablation: Rank direction R11 is highest-scoring in all 500 examples, while R7 is consistently among the lowest-scoring directions.
  • Diagnostic interventions: The study uses layer-wise adapter removal and direct logit attribution on CareBench examples, scoring reference strings with length-normalized teacher-forced log-probability.

D.2 Layer-Wise Adapter Removal

Layer-wise diagnostics show that FRAMES2LORA’s functional effect is concentrated more strongly in later transformer layers than its generated-weight norms alone would suggest. Representation shifts become increasingly aligned with the diagnostic answer direction near the output logits, while both attention and MLP components contribute.

  • Layer-wise adapter removal: Negative removal effects identify layers whose generated LoRA updates improve the diagnostic score.Values near zero indicate little measurable effect from removing a layer.
  • Layer-wise adapter removal: Later-layer updates produce larger negative removal effects than several early-layer updates despite sometimes smaller weight norms.This indicates that adapter use is not uniform across the transformer stack.
  • Layer-wise adapter removal: Generated-weight norm is not a complete measure of adapter importance because large updates can have weak functional effects.The authors interpret this as a norm–function dissociation rather than a full causal explanation.
  • Direct logit attribution: Adapter-induced representation shifts are weakly aligned with the diagnostic direction early and mid-network but become more aligned in later layers.The pattern places the strongest direct alignment close to the output logits.
  • Direct logit attribution: Both attention and MLP shifts contribute to the late-stage steering pattern.The authors avoid assigning the adapter a specific memory mechanism and instead describe representation changes throughout the network.

E Training Details

FRAMES2LORA training uses frozen video and answer models while optimizing only the hypernetwork, with matched configurations across the 500M and 2.2B scales. The evaluation materials include score distributions, per-example differences, token-F1 distributions, and QA and captioning spider plots, alongside qualitative benchmark examples.

  • Training configuration: Only the hypernetwork parameters are trained; the video encoder and answer model remain frozen.This configuration applies to both FRAMES2LORA model scales.
  • Training configuration: Both scales use rank-16 adapters, 12 uniformly sampled frames, 384-pixel maximum video dimension, and Perceiver latent size 512.The shared settings also include learning rate 1 × 10−4, warmup ratio 0.03, and weight decay 0.01.
  • Training configuration: The 500M run uses 4 A100 GPUs for 37 wall-clock hours, while the 2.2B run uses 6 A100 GPUs for 201 wall-clock hours.Effective batch sizes are 384 and 240, respectively.
  • Evaluation: Evaluation reports LLM-judge and token-F1 score distributions with corresponding per-example differences between the direct baseline and FRAMES2LORA.The referenced plots are Figures 9–12.
Loading 2606.04351v2…