Source-linked AI summary
NVILA: Efficient Frontier Visual Language Models
Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, Xiuyu Li, Yunhao Fang, Yukang Chen, Cheng-Yu Hsieh, De-An Huang, An-Chieh Cheng, Vishwesh Nath, Jinyi Hu, Sifei Liu, Ranjay Krishna, Daguang Xu, Xiaolong Wang, Pavlo Molchanov, Jan Kautz, Hongxu Yin, Song Han, Yao Lu
TL;DR
VLM accuracy has advanced, but efficiency across training, fine-tuning, and deployment remains less studied despite substantial resource demands. NVILA uses scale-then-compress visual processing and lifecycle-wide optimizations, matching or surpassing leading open and proprietary VLMs while reducing training and inference costs.
Problem
VLM efficiency in training, fine-tuning, and deployment has received much less attention despite high computational, memory, and latency demands.
Method
NVILA scales spatial and temporal resolutions, compresses visual tokens, and optimizes efficiency across training, fine-tuning, and deployment.
Results
NVILA matches or surpasses leading open and proprietary VLMs across image and video benchmarks while reducing training cost by 1.9–5.1×.
Takeaways & Limitations
NVILA demonstrates that high-resolution image and long-video processing can be combined with substantial resource efficiency in an open VLM family.
Takeaways & Limitations
Alternative learnable visual token compressors performed no better than simple spatial-to-channel compression, which the authors attribute to optimization difficulty.
Abstract
from arXiv · showhide
Visual language models (VLMs) have made significant advances in accuracy in recent years. However, their efficiency has received much less attention. This paper introduces NVILA, a family of open VLMs designed to jointly optimize efficiency and accuracy. Building on top of VILA, we improve its model architecture by first scaling up the spatial and temporal resolutions, and then compressing visual tokens. This "scale-then-compress" approach enables NVILA to efficiently process high-resolution images and long videos. We further conduct a systematic investigation that enhances NVILA's efficiency throughout its entire lifecycle, from training and fine-tuning to deployment. NVILA matches or surpasses the accuracy of leading open and proprietary VLMs across a wide range of image and video benchmarks. At the same time, it reduces training cost by 1.9-5.1x, prefilling latency by 1.6-2.2x, and decoding latency by 1.2-2.8x. We release our code and models to facilitate reproducibility.
1. Introduction
VLM accuracy has advanced across diverse applications, but efficiency in training, fine-tuning, and deployment remains comparatively underexplored. NVILA addresses this gap with a scale-then-compress architecture and lifecycle-wide efficiency optimizations while maintaining strong benchmark accuracy.
- VLM research has improved accuracy and broadened applications, while comparatively less attention has been paid to efficiency.
- Training a state-of-the-art 7B VLM can take up to 400 GPU days, creating a significant entry barrier for researchers.Fully fine-tuning a 7B VLM can require over 64 GB of GPU memory, while edge deployment is latency-sensitive.
- NVILA jointly optimizes efficiency and accuracy as a family of open VLMs built on VILA.
- Its scale-then-compress strategy first increases spatial and temporal resolution, then compresses visual tokens to process high-resolution images and long videos efficiently.The paper also optimizes efficiency across training, fine-tuning, and deployment.
- 1.9–5.1× lower training cost, 1.6–2.2× lower prefilling latency, and 1.2–2.8× lower decoding latency are reported for NVILA.NVILA matches or surpasses leading open and proprietary VLMs across image and video benchmarks.
2. Approach
The approach improves VLM efficiency through a scale-then-compress architecture and complementary optimizations across training, fine-tuning, and deployment.
- NVILA first scales spatial and temporal resolutions to raise the accuracy ceiling, then compresses visual tokens to recover efficiency.
- The design targets VLMs that are both highly accurate and efficient.
- Efficiency analysis covers the model lifecycle from training through fine-tuning and deployment, with analyses generally using the 8B model.
2.1. Efficient Model Architecture
NVILA combines adaptive high-resolution visual processing with spatial and temporal token compression. This raises visual detail and benchmark accuracy while controlling token counts and computational cost.
- NVILA uses a visual encoder, projector, and token processor, with SigLIP, a two-layer MLP, and Qwen2 respectively.Images undergo Dynamic-S2 tiling and spatial compression before visual and text tokens enter the language model.
- VILA’s fixed image resolution and limited video sampling lose information, especially on text-heavy image and long-video benchmarks.
- Scaling resolution and frame count raises the accuracy ceiling but increases visual tokens and compute, motivating subsequent compression.Doubling resolution quadruples visual tokens, while LLM self-attention further amplifies cost.
- Dynamic-S2 adaptively tiles images while preserving aspect ratio at the largest scale instead of uniformly resizing every image to a square.
- Up to 30 points of absolute accuracy gain on text-heavy benchmarks is reported with Dynamic-S2.
- A 2×2 spatial-to-channel reshape cuts token count 4× without sacrificing accuracy, whereas 3×3 compression can cause a nearly 10-point DocVQA drop.
- An additional visual encoder pre-training stage recovers most of the 3×3 compression accuracy loss while preserving a 2.4× speedup over the 2×2 baseline.
- Alternative learnable compressors, including TokenLearner and Perceiver Resampler, perform no better than simple spatial-to-channel compression at the same reduction ratio.The authors attribute this to optimization difficulty and leave deeper investigation for future work.
2.2. Efficient Training
NVILA reduces training cost through dataset pruning and FP8 mixed-precision training while addressing variable-length VLM workloads. DeltaLoss selects informative data, and FP8 accelerates training without degrading accuracy.
- Dataset Pruning: DeltaLoss pruning outperforms other data-selection methods, with minimal accuracy loss at a 50% keep-ratio.
- Dataset Pruning: Indiscriminate Pixmo dataset growth can degrade DocVQA and TextVQA, whereas DeltaLoss pruning yields consistent benchmark improvements even at small keep-ratios.
- FP8 Training: FP8 mixed-precision training accelerates NVILA without degrading accuracy, with the largest gains when gradient checkpointing is disabled.
- FP8 Training: VLM batches vary from hundreds to tens of thousands of tokens, so batches dominated by short samples can underutilize the GPU.
2.3. Efficient Fine-Tuning
NVILA improves VLM fine-tuning efficiency by assigning separate learning rates to the vision and language components and tuning only a small subset of vision parameters. This preserves competitive accuracy while reducing memory, time, and compute requirements.
- Fine-Tuning Recipe: The ViT learning rate should be 5–50× smaller than the LLM learning rate during joint PEFT fine-tuning.
- Fine-Tuning Recipe: Tuning only the ViT LayerNorm layers matches LoRA accuracy while reducing vision-encoder training time by 25%.
- Fine-Tuning Recipe: NVILA can be fine-tuned on downstream tasks within 24 GB of GPU memory while maintaining on-par performance.
- Fine-Tuning Recipe: The recommended recipe uses LoRA or QLoRA for the LLM and a much smaller learning rate for the ViT LayerNorm layers.
2.4. Efficient Deployment
NVILA deploys a phase-specific inference engine that compresses visual tokens and quantizes the vision tower and language backbone differently. These choices target compute-bound prefilling and memory-bound decoding separately.
- Inference Optimization: Token compression reduces the LLM backbone workload, leaving the vision tower responsible for over 90% of prefilling latency.
- Inference Optimization: W8A8 quantization targets the vision tower for lower time-to-first-token, while W4A16 quantization targets the LLM backbone during decoding.
- Inference Optimization: FP16 accumulation in W4A16 GEMM kernels provides a 1.7× kernel speedup without compromising accuracy.
- Inference Optimization: Together, W4A16 on the LLM and W8A8 on the ViT reduce time-to-first-token by 28%.
3. Experiments
NVILA uses a staged training pipeline and evaluates two model variants across broad image and video benchmarks. It combines scale-then-compress architecture with full-stack optimizations to retain strong accuracy while improving efficiency.
- Training and Models: NVILA trains through five stages, including visual encoder pre-training to recover compression-related accuracy loss and video instruction-tuning for long-video understanding.
- Training and Models: NVILA-Lite maximizes efficiency, while NVILA trades a small amount of efficiency for higher accuracy; both share the same training pipeline.
- Image Benchmarks: NVILA-15B reaches 94.1 on AI2D, approaching GPT-4o at 94.2, Gemini 1.5 Pro at 94.4, and Claude 3.5 Sonnet at 94.7.
- Video Benchmarks: NVILA-8B achieves state-of-the-art results across evaluated video benchmarks among open-source 7–8B models and matches GPT-4o mini.
- Efficiency Results: A 3×3 spatial-to-channel reshape yields a 2.4× speedup with little accuracy loss, while full-stack optimizations improve training, fine-tuning, and inference efficiency.
- Efficiency Results: With 64 sampled frames on one RTX 4090, NVILA achieves up to 2.2× prefilling speedup and up to 2.8× higher decoding throughput over Qwen2-VL.
4. More Capabilities
NVILA extends VLM capabilities to temporal localization, robotic navigation, and medical imaging while maintaining strong performance and efficient inference. These applications cover video understanding, embodied agents, and precision-critical medical tasks.
- Temporal localization: NVILA adds temporal localization using discrete time tokens and smoothed cross-entropy training, substantially outperforming all baselines across reported metrics.The evaluation uses the same data mixture as VILA-1.5 while replacing its base VLM with NVILA.
- Robotic navigation: NVILA supports Vision-Language Navigation by processing language instructions and multi-frame observations to plan actions over successive states.The NaVILA framework fine-tunes NVILA with navigation-specific supervised data curated from a simulator.
- Robotic navigation: The real-world camera→GPU→action pipeline runs at 1 Hz, demonstrating NVILA’s suitability for real-time robotic deployment on resource-constrained edge devices.The robot platform uses a Unitree Go2 with LiDAR and an Intel RealSense camera, while an RTX 4090 processes eight frames per action step.
- Medical application: NVILA-M3 combines domain-expert models with vision-language learning for medical segmentation, classification, and related tasks.The framework targets intricate medical features and nuanced relationships between visual inputs and textual annotations.
- Medical application: NVILA-M3 achieves an overall 9% improvement over task-specific state-of-the-art baselines, supporting the value of combining specialized expertise with general-purpose VLMs.The reported medical metrics include VQA accuracy, BLEU-4 and ROUGE for report generation, and F1 for classification.
- Inference efficiency: Against Qwen2-VL-7B, NVILA achieves 1.6–2.2× faster prefilling and up to 2.8× higher decoding throughput on image and video tasks.The comparison uses Qwen2-VL with W4A16 quantization and NVILA with a specialized inference engine on one RTX 4090 GPU.
5. Related Work
Related work has rapidly improved VLM accuracy and applications, but efficiency across training, fine-tuning, and deployment remains less studied. NVILA addresses gaps in token reduction, data selection, low-precision training, quantization, and parameter-efficient fine-tuning.
- VLM progress: Proprietary and open VLMs have advanced substantially, with open models progressively narrowing the gap to proprietary systems on key benchmarks.Representative open models include InternVL2, Qwen2-VL, LLaVA-OneVision, Llama 3.2 Vision, Molmo, NVLM, and MiniCPM-V.
- Efficiency gap: Despite accuracy gains, efficiency in VLM training, fine-tuning, and deployment has received much less attention.The paper frames end-to-end efficiency as the central unresolved focus.
- Token reduction: Prior token-reduction work studies spatial and temporal compression, but not token reduction for frontier VLMs where preserving accuracy at scale is central.This motivates NVILA’s scale-then-compress design.
- Data selection: VLM data-selection research remains limited compared with LLM research, and this work specifically targets supervised fine-tuning data for VLMs.Earlier approaches include domain mixing, samplewise selection, and theory-driven optimal selection for LLM pre-training.
- Low-precision training: FP8 training has gained traction for LLMs, but prior work had not demonstrated its feasibility for VLMs without sacrificing accuracy.
- Model compression: Best practices remain open for pruning, distillation, and quantization across the assembled VLM and its constituent language and vision components.Quantizing the vision encoder becomes especially important at higher input resolutions and longer video lengths.
- Parameter-efficient fine-tuning: Efficient fine-tuning methods are widely used for LLMs, but remain underexplored for VLMs that combine a vision encoder with an LLM.NVILA addresses this gap as part of its lifecycle-wide efficiency investigation.
6. Conclusion
NVILA balances efficiency and accuracy through scale-then-compress processing of high-resolution images and long videos. Its lifecycle-wide optimization supports strong benchmark performance and applications spanning temporal localization, robotics, and medical imaging.
- Conclusion: NVILA’s scale-then-compress paradigm processes high-resolution images and long videos efficiently while maintaining high accuracy.The design first scales spatial and temporal resolutions, then compresses visual tokens.
- Conclusion: NVILA systematically optimizes efficiency across training, fine-tuning, and deployment while matching or surpassing leading VLMs in accuracy.
- Conclusion: The resulting model family supports applications including temporal localization, robotic navigation, and medical imaging.