Source-linked AI summary
InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models
Hongyuan Tao, Bencheng Liao, Shaoyu Chen, Haoran Yin, Qian Zhang, Wenyu Liu, Xinggang Wang
TL;DR
Long multimodal inputs challenge Transformers’ compute and memory budgets, while linear VLMs can lose high-frequency visual details. InfiniteVL combines Gated DeltaNet with Full Attention and adapts attention into sparse mechanisms for long contexts. Its base and specialized variants report Transformer-level performance alongside decoding, prefill, and streaming-efficiency results.
Problem
Long multimodal inputs strain Transformer computation and memory, while linear VLMs can lose high-frequency visual details needed for precise perception.
Source: §1 · p. 2, §2.2 · p. 3Method
InfiniteVL interleaves Gated DeltaNet with Full Attention, then fine-tunes dense attention into vision-specific sparse mechanisms for online and offline long-context use.
Source: §1 · p. 2Results
InfiniteVL-Base achieves over 1.7× decoding speedup with Transformer-level fundamental multimodal understanding; InfiniteVL-Offline reports 5× prefill acceleration at 256K, and InfiniteVL-Online sustains 25 FPS with constant O(1) memory.
Source: §1 · pp. 2–3Takeaways & Limitations
The specialized variants target offline retrieval and online streaming while balancing long-context efficiency with precise visual perception.
Source: §5 · p. 11
Abstract
from arXiv · showhide
Vision-Language Models (VLMs) are increasingly tasked with ultra-long multimodal understanding. While linear architectures offer constant computation and memory footprints, they often struggle with high-frequency visual perception compared to standard Transformers. To bridge this gap, we introduce \textbf{InfiniteVL}. We first develop a hybrid base model called \textbf{InfiniteVL-Base} that interleaves a small fraction of Full Attention layers with Gated DeltaNet. Empowered by a tailored distillation and fine-tuning strategy, InfiniteVL-Base matches the fundamental multimodal performance of equivalent Transformers while achieving a \textbf{1.7$\times$} decoding speedup. However, the quadratic complexity of the retained Full Attention inevitably becomes an efficiency bottleneck when scaling to ultra long context. To break this barrier, we propose a novel Long-Sequence Architectural Fine-Tuning strategy that seamlessly transforms the dense attention into vision-specific sparse mechanisms. This yields two specialized variants: \textbf{InfiniteVL-Offline} for offline retrieval and \textbf{InfiniteVL-Online} for online streaming. By eliminating the computation explosion of global attention without sacrificing high-frequency visual recall, InfiniteVL-Offline achieves Transformer-level length generalization with a \textbf{5x} prefill acceleration at 256K context. Concurrently, InfiniteVL-Online delivers robust streaming perception with a constant memory footprint and a real-time throughput of \textbf{25} FPS. Code and models are available at https://github.com/hustvl/InfiniteVL.
1 Introduction
Long multimodal inputs strain Transformers’ computation and memory, while linear VLMs can lose high-frequency visual detail. InfiniteVL combines Gated DeltaNet and Full Attention, then adapts attention for online and offline long-context use.
1 Introduction
- Transformers’ quadratic self-attention and growing KV cache make long multimodal inputs costly in computation and memory, limiting extended video understanding and continuous agent interaction.The passage also notes deployment pressure on edge devices requiring real-time performance and low memory.Source: §1 · p. 2
- Linear attention offers constant computation and memory, but its state compression can discard high-frequency visual details needed for OCR and high-resolution document comprehension.This creates a trade-off between long-term efficiency and precise visual performance.Source: §1 · p. 2
- InfiniteVL-Base allocates about 75% of its layers to Gated DeltaNet and 25% to Full Attention, with distillation pretraining followed by Continuous SFT.The hybrid achieves multimodal understanding on par with mainstream Transformers and expands context-length processing capabilities by 3.1×.Source: §1 · p. 2
- Long-Sequence Architectural Fine-Tuning converts dense attention into vision-specific sparse mechanisms, producing InfiniteVL-Online for streaming and InfiniteVL-Offline for offline comprehension.Online combines a sliding window and attention sinks; offline uses dynamic top-k chunk-level retrieval.Source: §1 · p. 2
- InfiniteVL-Base delivers over a 1.7× decoding speedup while achieving Transformer-level fundamental multimodal understanding; InfiniteVL-Offline adds 5× prefill acceleration at 256K, and InfiniteVL-Online sustains 25 FPS with constant O(1) memory.The base result is reported alongside the long-context results for the two specialized variants.Source: §1 · pp. 2–3
2 Related Work
Long-context VLMs face a tension between scaling multimodal understanding and retaining precise visual details. Linear and hybrid attention address parts of this challenge, but existing multimodal hybrids retain performance and efficiency gaps.
2.1 Vision-Language Models
- Standard Transformers are constrained on long sequences by quadratic computation and a growing KV cache, while cache dropping and sliding windows can compromise long-term memory and situational coherence.Long-term multimodal understanding is relevant to embodied AI and physical scene perception.Source: §2.1 · p. 3
2.2 Linear Attention
- Linear attention and state-space models compress history into fixed-size states for linear complexity and constant-memory inference, but linear VLMs can lose high-frequency visual details and underperform on OCR and document understanding.The cited methods include Mamba, GLA, and Gated DeltaNet; the visual-detail limitation motivates balancing compression with local detail retention.Source: §2.2 · p. 3
2.3 Hybrid Architecture
- Hybrid architectures reintroduce a small fraction of Full Attention to recover precise features, but existing hybrid VLMs still show a performance gap against leading Transformer VLMs and retain a long-sequence efficiency bottleneck.Prior hybrid work in LLMs has found that even a minimal proportion of Full Attention can bridge retrieval gaps in linear architectures.Source: §2.3 · p. 3
3 Method
InfiniteVL combines a vision encoder and LLM backbone with interleaved Full Attention and Gated DeltaNet, then fine-tunes the architecture for ultra-long contexts. The base model uses attention for precise retrieval and linear layers for memory compression.
3 Method
- InfiniteVL first builds a hybrid base model, then adapts global attention into vision-specific sparse mechanisms for online streaming or offline contexts.The training strategy proceeds through distillation pretraining, Continuous SFT, and long-sequence architectural fine-tuning.Source: §3 · p. 4, Figure 3 · p. 5
3.1 InfiniteVL-Base
- InfiniteVL-Base uses a Qwen2.5-VL vision encoder, projection MLP, and decoder-only backbone with nine hybrid blocks, each containing one Full Attention layer and three Gated DeltaNet layers.The layers use residual pathways and Pre-Layer Normalization with SiLU activations; the Full Attention layer uses GQA, and spatial-temporal semantics use 3D RoPE.Source: §3.1 · p. 4
- The model’s standard Full Attention computes outputs from queries, keys, and values using softmax-scaled query-key similarities.This is the attention operation used in the Full Attention layer.Source: §3.1 · p. 4
- Gated DeltaNet compresses memory into a fixed-size state, with a data-dependent decay gate and an update-rate control governing the recurrent memory update and retrieval.The layer also uses a 1D convolution with window size 4 and an output gate.Source: §3.1 · p. 4
3.2 Efficient Multi-Stage Training Strategy
InfiniteVL-Base uses a progressive training pipeline to retain pretrained multimodal knowledge while stabilizing newly initialized linear modules. The pipeline combines distillation pretraining and Continuous SFT, using a curated multimodal corpus.
3.2 Efficient Multi-Stage Training Strategy
- A progressive two-stage pipeline first stabilizes newly initialized linear modules with Distillation Pretraining, then uses Continuous SFT for component alignment and multimodal interactions.The strategy is designed to preserve pretrained weights and constrain the overall training budget.Source: §3.2 · p. 5
3.2.1 Training Data.
- The training corpus combines FineVision, LLaVA-OneVision-1.5, PixMo, The Cauldron, and Docmatix, spanning tasks from OCR to mathematics.Its scale and diversity are described as matching leading VLMs such as InternVL2.5.Source: §3.2.1 · p. 5
3.2.2 Distillation Pre-training.
- Distillation replaces selected Qwen2.5-VL attention layers with Gated DeltaNet and aligns the modules using same-input layerwise feature matching followed by end-to-end token-logit KL divergence.The layerwise objective minimizes MSE between teacher and student layer outputs; the end-to-end objective minimizes KL divergence between token-level logits.Source: §3.2.2 · p. 5
3.2.3 Continuous SFT.
- Continuous SFT fine-tunes with cross-entropy to unlock linear modules and mitigate exposure bias, increasing maximum image resolution to 1344 × 1344 while retaining an 8192-token input limit.The loss uses the target distribution and the student’s token probabilities.Source: §3.2.3 · p. 6
3.3 Long-Sequence Architectural Fine-Tuning Strategy
Long-sequence architectural fine-tuning converts retained dense attention into vision-specific sparse mechanisms for offline comprehension and online streaming. The variants target offline prefill and retrieval costs or streaming memory growth while retaining the linear layers’ history compression.
3.3 Long-Sequence Architectural Fine-Tuning Strategy
- The strategy addresses Full Attention’s quadratic prefill-time and linear decoding-memory growth while preserving precise visual perception and avoiding catastrophic forgetting of pretrained knowledge.Offline comprehension is bottlenecked by upfront O(L^2d) prefill and exact retrieval needs; streaming instead requires bounded memory.Source: §3.3 · p. 6, §3.3.1 · p. 6, §3.3.2 · p. 6
3.3.1 InfiniteVL-Offline for Offline Comprehension.
- InfiniteVL-Offline complements linear history compression with sparse retrieval, reducing prefill complexity from O(L^2d) to O(L·Cd) for retrieved blocks of total length C ≪ L.Queries route to top-k relevant 32-token chunks while retaining initial prompts and local temporal context.Source: §3.3.1 · p. 6
3.3.2 InfiniteVL-Online for Online Streaming.
- Online streaming requires constant memory because an unbounded frame influx makes standard attention’s O(Ld) KV cache eventually exceed GPU VRAM.Naive sliding-window attention can discard early system instructions and scene setups that help stabilize the softmax distribution.Source: §3.3.2 · pp. 6–7
- InfiniteVL-Online retains initial attention sinks and a sliding window, bounding its KV cache to 256 sink tokens plus a 2,048-token window.This reduces decoding memory from O(Ld) to O(1) and makes per-token decoding time constant.Source: §3.3.2 · p. 7
3.3.3 Long-Sequence Fine-Tuning.
- Stage III fine-tuning adapts both variants’ inherited InfiniteVL-Base weights to sparse masks using SFT data and video QA samples, targeting length generalization up to 32,768 tokens.The training blend contains 800,000 Stage II SFT samples and 200,000 video QA pairs from LLaVA-Video-178K.Source: §3.3.3 · p. 7
4 Experiments
Experiments report foundational parity with Transformer baselines and show that specialized long-context variants preserve multimodal performance while reducing decoding, prefill, or streaming costs. Ablations examine the linear module, attention mix, and staged training strategy.
4.1.1 Baselines.; 4.1.2 Benchmarks.
- The experiments use an equivalent-scale Qwen2.5-VL* Transformer trained with the same pipeline and data, and evaluate foundational plus extended-context benchmarks.Long-context comparisons apply the same fine-tuning recipe to Qwen2.5-VL* and InfiniteVL-Base.Source: §4.1.1 · pp. 7–8, §4.1.2 · p. 8
4.2.1 Foundational Multimodal Performance.
- InfiniteVL-Base achieves consistent parity with the controlled Qwen2.5-VL* baseline and leading similar-scale models, including on visually intensive OCR, chart, and document tasks.The evaluation spans 13 general benchmarks and extended-context high-resolution image, offline video, and online streaming tasks.Source: §4.2.1 · p. 8, §4.1.2 · p. 8
4.2.2 Efficiency Analysis.
- At 450K context, InfiniteVL-Base achieves a 1.7× per-token speedup and reaches 1,589K tokens before OOM, versus 454K for Qwen2.5-VL-3B.At 256K context, Base also delivers a 2.8× prefill acceleration over Qwen2.5-VL-3B.Source: §4.2.2 · p. 9, Figure 5 · p. 9
4.3.1 Multimodal Long-context Performance.
- Offline matches InfiniteVL-Base across evaluated benchmarks and context lengths, following Hybrid/Transformer length-extrapolation trajectories; Online matches or exceeds Base on StreamingBench and degrades less than SWA.The comparisons cover high-resolution images, offline long-video comprehension, and online streaming perception.Source: Figure 4 · p. 8, Table 2 · p. 8, §4.3.1 · p. 9
4.3.2 Offline and Online Video Understanding Efficiency.
- At 256K input length, InfiniteVL-Offline achieves 5× prefill speedup over Qwen2.5-VL*; CUDA Graph acceleration brings InfiniteVL-Online streaming throughput to 25 FPS.Without CUDA Graph, Online sustains 8 FPS; unconstrained KV-cache growth makes Base and Qwen2.5-VL* slow down and eventually run out of memory.Source: §4.3.2 · p. 9, Figure 6 · p. 10
4.4.1 Linear Layer Selection.
- Gated DeltaNet improves training stability and downstream performance, particularly on DocVQA and OCRBench, whereas Mamba and GLA show limited performance on information-dense document and text tasks.Vanilla Linear Attention is unstable and does not inherit the teacher’s pretrained knowledge.Source: §4.4.1 · pp. 10–11
4.4.2 Mixing Strategy.
- A Full Attention-to-linear-layer ratio of 1:3 achieves parity with the Transformer baseline across comprehensive benchmarks, while even a minimal Attention proportion boosts visually intensive performance..Source: §4.4.2 · p. 11
4.4.3 Training Strategy.
- Stage I distillation is necessary: training from scratch with SFT alone yields extremely poor results, and omitting Stage III leaves Base behind on multimodal long-context tasks.After Stage II continuous SFT, the model surpasses the teacher across multiple metrics.Source: §4.4.3 · p. 11
5 Conclusion
InfiniteVL combines constant-memory Gated DeltaNet with a fraction of Full Attention to balance visual precision and long-context efficiency. Its specialized variants target offline comprehension and online streaming, with experiments reporting Transformer-baseline capability, faster prefill, and real-time streaming.
5 Conclusion
- InfiniteVL matches strictly controlled Transformer baselines while delivering 5× prefill speedup at 256K context and stable real-time streaming at 24 FPS.InfiniteVL-Offline targets precise offline long-video comprehension, while InfiniteVL-Online targets constant-memory streaming.Source: §5 · p. 11
Appendix
The appendix adds material omitted from the main paper, including cache-evolution analysis, reproduction details, and visualization cases.
Appendix
- Appendix sections analyze linear-layer cache evolution under long-term streaming, supplement training and evaluation setup, and show VQA, text-rich, and ultra-long streaming examples.Source: Appendix · p. 16
6 Analysis of Cache Evolution under Streaming Inputs
The analysis tracks the linear-layer memory cache across streaming frames and finds its norm stabilizes after an initial rise, without unbounded growth.
6 Analysis of Cache Evolution under Streaming Inputs
- Each linear-layer output multiplies the query by a memory cache formed from the cumulative outer product of keys and values.The cache's stability directly affects output stability.Source: §6 · p. 16
- The tracked memory cache has dimensions 16 × 128 × 256, and its L2 norm is measured as input frames increase.Source: §6 · p. 16
- The cache norm rises rapidly in the initial frames and then stabilizes, showing no unbounded growth during streaming.The authors interpret this as evidence of stable cache management and comprehension capability for unlimited streaming inputs.Source: §6 · p. 16, Figure 7 · p. 16
7 Comprehensive Training and Evaluation Setup Details
The section specifies the training hardware, data and three-stage training regimen, as well as the evaluation configuration and benchmarks.
7 Comprehensive Training and Evaluation Setup Details
- Training uses NVIDIA H20 GPUs with BF16 precision and approximately 10M samples from publicly available, open-source datasets.The required data portion is uniformly subsampled at each stage; optimization uses AdamW, cosine annealing, and a 5% linear warm-up.Source: §7 · p. 16
- Stage I trains layer-wise and end-to-end distillation on 1M multimodal Caption & QA pairs each; Stage II uses 8M multimodal QA pairs; Stage III combines 0.25M long-sequence Caption & QA pairs with 0.75M Stage II-corpus SFT data.Stage I uses a 2 × 10−4 learning rate and batch size 64; Stage II uses 5 × 10−5 and batch size 256; Stage III uses 2 × 10−5 and batch size 64.Source: §7 · p. 17
- Evaluation uses VLMEvalKit defaults, a 256-token generation limit, and image resolutions from 128 × 128 to 1344 × 1344.Visual tokens are prepended to prompts; Qwen3-4B-Instruct evaluates MMMU and MathVista-MINI reasoning tasks.Source: §7 · p. 17
8 Case Study across multiple scenarios
The case studies cover fundamental vision-language understanding and long-term streaming, with reported comprehension and reasoning maintained beyond 512K tokens.
8 Case Study across multiple scenarios
- InfiniteVL maintains stable comprehension and reasoning in streaming examples with contexts exceeding 512K tokens.The examples use two extended street-view and dashcam videos, sampled at 1 FPS, with questions posed at randomly selected timestamps.Source: §8 · p. 17
- Figure 8 examples are described as showing comprehension in high-resolution, text-intensive, and complex-structure scenarios.Source: §8 · p. 17, Figure 8 · p. 18
- The streaming case study updates the model's memory cache continuously as video frames arrive and tests questions at selected timestamps.The cited examples include street-view recordings and dashcam footage sampled at 1 FPS.Source: §8 · p. 17, Figure 9 · p. 19, Figure 10 · p. 20