PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision-Language Models for Edge Networks: A Comprehensive Survey

Ahmed Sharshar, Latif U. Khan, Waseem Ullah, Mohsen Guizani

arXiv:2502.07855v2cs.CVcs.AIcs.CL

TL;DR

VLM deployment on edge devices is limited by memory, computational, energy, and cross-domain generalization constraints. This survey synthesizes compression, lightweight architectures, training, fine-tuning, deployment, and application strategies, concluding that practical adoption requires efficient, adaptable, and interoperable systems.

  • Problem

    VLMs face high computational demands, large data requirements, modality-alignment gaps, memory inefficiency, and limited generalization on resource-constrained edge devices.

  • Method

    The survey reviews lightweight architectures, pruning, quantization, knowledge distillation, efficient training and fine-tuning, deployment strategies, and application-specific approaches for edge VLMs.

  • Results

    The survey synthesizes design strategies and solution approaches spanning model compression, dynamic and modular deployment, context-aware systems, and lightweight multimodal applications.

  • Takeaways & Limitations

    Practical edge adoption depends on lightweight models that support real-time multimodal processing, continual learning, resource-aware adaptation, and privacy-preserving deployment.

  • Takeaways & Limitations

    Further work is needed on interoperability for massive edge deployment and on robustness, adaptability, and generalization across diverse domains.

Abstract

from arXiv · show

Vision Large Language Models (VLMs) combine visual understanding with natural language processing, enabling tasks like image captioning, visual question answering, and video analysis. While VLMs show impressive capabilities across domains such as autonomous vehicles, smart surveillance, and healthcare, their deployment on resource-constrained edge devices remains challenging due to processing power, memory, and energy limitations. This survey explores recent advancements in optimizing VLMs for edge environments, focusing on model compression techniques, including pruning, quantization, knowledge distillation, and specialized hardware solutions that enhance efficiency. We provide a detailed discussion of efficient training and fine-tuning methods, edge deployment challenges, and privacy considerations. Additionally, we discuss the diverse applications of lightweight VLMs across healthcare, environmental monitoring, and autonomous systems, illustrating their growing impact. By highlighting key design strategies, current challenges, and offering recommendations for future directions, this survey aims to inspire further research into the practical deployment of VLMs, ultimately making advanced AI accessible in resource-limited settings.

I. INTRODUCTION

VLMs combine visual and language understanding for diverse applications, but their deployment on edge devices is constrained by memory, computation, energy, and heterogeneous hardware. The survey reviews compression, efficient architectures, deployment strategies, and application domains addressing these constraints.

  • VLMs support tasks including image captioning, visual question answering, and visual content generation across domains such as surveillance and autonomous navigation.
  • Pruning, quantization, knowledge distillation, and specialized hardware reduce model size or computational overhead for more efficient edge inference.Pruning removes redundant parameters, quantization lowers weight and activation precision, and distillation transfers knowledge from a larger teacher to a smaller student.
  • Edge deployment is difficult because current VLMs require substantial memory, computation, and energy beyond typical device capabilities.GPT-3 requires about 350 GB of inference memory, while edge devices commonly have 1,000–5,000 mAh backup energy capacity.
  • Deployment must accommodate heterogeneous device capabilities through dynamic inference, model scaling, and architectures designed for lower computational requirements.These approaches balance resource use and accuracy by adapting model complexity or simplifying attention mechanisms and network layers.
  • Lightweight VLMs are being applied to healthcare, environmental monitoring, retail, security, and other real-time edge scenarios.Examples include portable medical image analysis, zero-shot environmental surveys, smart shopping assistants, and privacy-aware surveillance.
  • The survey extends prior reviews with a comprehensive overview of VLM design, deployment challenges, open research problems, and solution approaches.

D. Our Survey

The survey examines VLM techniques, architectures, applications, edge deployment, privacy, and security, while presenting open challenges and solution approaches. It also explains how textual and visual representations are embedded, encoded, and fused in multimodal models.

  • The survey presents key concepts, design aspects, and high-level architectures for Vision-Language Models.
  • It covers extending VLMs from cloud to edge through efficient training and fine-tuning, deployment considerations, and privacy and security issues.
  • VLMs process textual and visual information through tokenization, embedding, encoding, and cross-modal alignment for tasks including captioning, retrieval, and question answering.
  • Text Representation: Text processing maps tokens into word embeddings, adds positional embeddings, and passes the combined representations through transformer layers.
  • Model Architectures: VLM architectures use separate visual and textual streams or a unified encoder, with modality fusion occurring through interactions or within the shared architecture.

B. Mechanisms for Vision-Language Interaction

VLMs integrate visual and textual embeddings through either single-stream fusion or dual-stream encoders. Single-stream models simplify integration and support fine-grained alignment, while dual-stream models separately process modalities before cross-modal integration, increasing computational complexity.

  • Single-Stream Architecture: Single-stream models interleave visual and textual embeddings into one sequence processed by a shared transformer.Each token can attend to and influence other tokens, supporting fine-grained multimodal alignment.
  • Single-Stream Architecture: Single-stream architectures reduce memory and inference requirements by using one encoder, but concatenated longer sequences increase computational burden.They are useful for tasks such as image captioning, VQA, and grounded dialogue, while potentially producing non-ideal performance.
  • Dual-Stream Architecture: Dual-stream models use independent visual and textual encoders before merging representations through cross-attention or another integration mechanism.Separate processing can suit modalities with limited overlapping features and complex cross-modal relationships.
  • Dual-Stream Architecture: Dual-stream architectures can support detailed image-text processing for VQA and retrieval, but require multiple encoders and additional modality-alignment computation.Their separate streams may outperform single-stream models on tasks requiring specialized processing of both modalities.

C. Existing Lightweight VLM Models

Existing lightweight VLMs reduce model size or inference cost through compact architectures, token reduction, distillation, pruning, or efficient multimodal fusion. They span general-purpose, task-specialized, and application-oriented systems, but robustness, adaptability, and generalization remain challenges.

  • Application-Oriented Models: Lightweight VLMs address varied applications, including image recognition, autonomous driving, instruction following, retrieval-augmented VQA, UI understanding, and consumer-device deployment.Examples include ViTamin, LINGO-2, InstructBLIP, RAVEN, ScreenAI, ALLaVA, and Xmodel-VLM.
  • Mobile and Lightweight Architectures: MobileVLM V2 reduces visual tokens with a Lightweight Downsample Projector and uses MobileLLaMA for fast multimodal processing on mobile devices.Its architecture is presented as an example of mobile-oriented VLM design.
  • Mobile and Lightweight Architectures: LightVLP uses a Gated Interactive Masked AutoEncoder and multimodal encoder to align visual and textual inputs with fewer parameters.The model emphasizes lightweight pre-training and efficient multimodal alignment.
  • Efficiency-Oriented Models: EfficientVLM uses knowledge distillation and modal-adaptive pruning, achieving 98.4% of teacher performance with 93 million parameters and 2.2× faster inference.The model targets tasks including visual question answering and image retrieval.
  • Efficiency-Oriented Models: MiniVLM combines a two-stage efficient feature extractor with transformer fusion, reducing model size by 73% and inference time by 94%.It retains 94–97% accuracy across various vision-language tasks and is described as suitable for edge applications.
  • Challenges and Future Directions: Lightweight VLMs still face challenges in robustness, adaptability, and generalization.Suggested future directions include adaptive learning, transfer learning, dynamic fusion, transparency, and interpretability for healthcare and autonomous driving.

D. Efficient Fine-Tuning Methods for Vision-Language Models

Efficient fine-tuning adapts large-scale VLMs to downstream tasks under limited computational budgets without requiring full retraining or full-model fine-tuning. The section centers on prompt-based and adapter-based methods.

  • Efficient Fine-Tuning: Efficient fine-tuning reduces the resource burden of retraining and full fine-tuning for downstream VLM tasks.The methods are designed for adapting large-scale models under limited computational budgets.
  • Efficient Fine-Tuning: The discussed research directions focus on prompt-based methods and adapter-based methods.These approaches form the section’s main categories of efficient fine-tuning.

1) Fine-tuning with Prompts:

Efficient VLM adaptation uses prompts, adapters, and hybrid methods to update few parameters while preserving pretrained weights. These techniques support multimodal alignment and edge-oriented efficiency across heterogeneous devices.

  • 1) Fine-tuning with Prompts:: Prompt-based fine-tuning shapes inputs to activate pretrained model capabilities with few parameter updates.Prompt tuning uses hard text templates or learned soft vectors, while prefix and P-tuning add trainable virtual tokens.
  • 1) Fine-tuning with Prompts:: Vision-language prompt methods such as DenseCLIP and ProDA align visual and textual features with performance comparable to full fine-tuning.
  • 2) Adapter-Based Fine-Tuning:: Adapter-based fine-tuning inserts lightweight task-specific modules while keeping the original model weights frozen.Adapter modules and LoRA reduce trainable parameters while maintaining performance across downstream tasks.
  • 2) Adapter-Based Fine-Tuning:: Parallel and task-specific adapters support multi-task learning and multimodal adaptation with reduced computational costs.AdapterFusion combines modules trained on different tasks, while VL-Adapter and Clip-Adapter target vision-language requirements.
  • 2) Adapter-Based Fine-Tuning:: APoLLo combines prompt learning and adapters for efficient, robust VLM fine-tuning through cross-modal attention between image and text pathways.It processes LLM-generated text and text-conditioned synthetic images as augmented inputs.
  • III. VLMS FOR EDGE NETWORKS: Edge computing processes and stores data locally to reduce latency and bandwidth use while improving privacy for IoT applications.Intelligent edge devices such as gateways and smart cameras support inference and analytics, using SoCs, GPUs, or specialized processors.
  • III. VLMS FOR EDGE NETWORKS: MobileVLM compresses visual features with a Lightweight Downsample Projector before combining them with textual features in MobileLLaMA.The architecture uses depthwise and pointwise convolutions to reduce visual-token processing before response generation.
  • III. VLMS FOR EDGE NETWORKS: Edge devices span diverse applications and hardware configurations, requiring trade-offs between computational time, energy efficiency, and task demands.SoCs suit IoT gateways, whereas GPUs or ASICs support heavier real-time image processing.

A. Existing Low Complexity VLMs

Low-complexity VLMs differ in whether they merely reduce model cost or are specifically designed for local, privacy-preserving edge inference. Representative systems use distillation, token compression, efficient pretraining, and hardware-aware design.

  • IoT combines connected sensing, transmission, and analysis with LLMs and VLMs to support automation, decision-making, and interaction.Its architecture includes perception, network, and application layers.
  • EdgeVL transfers CLIP teacher knowledge to a compact student using dual-modality distillation and quantization-aware contrastive learning without manual annotation.The framework supports RGB and non-RGB images for open-vocabulary edge classification.
  • Edge-specific VLMs add energy efficiency, low latency, and privacy-preserving inference constraints beyond lightweight model design.Lightweight models may still depend on cloud processing, whereas edge-specific models target local inference.
  • MiniVLM reduces visual feature extraction time by 95% compared to baseline models while retaining competitive downstream performance.
  • VILA combines interleaving data with instruction fine-tuning for efficient deployment across hardware such as Jetson Orin.It also emphasizes multimodal pretraining, in-context learning, and multi-image reasoning.
  • MobileVLM V2 uses LDPv2, pointwise and depthwise convolutions, and pooling to compress image tokens and reduce computational requirements.
  • EDGE-LLM addresses edge computational and memory overhead through efficient tuning and memory management while supporting continuous privacy-preserving adaptation.
  • Vision transformers for edge deployment use token pruning, quantization, and convolutional variants such as CvT and TinyViT for detection and segmentation.

B. Deployment of VLMs on Edge Devices

Edge deployment selects and compresses models according to data, hardware, latency, and accuracy constraints. The workflow combines preprocessing, model partitioning, compression, and adaptive evaluation to balance efficiency with performance.

  • 1) Data Selection and Pre-processing:: Edge deployment begins by collecting heterogeneous data and separating edge-appropriate inputs from cloud-appropriate inputs.Simpler data can be processed in real time locally, while complex data is assigned to resource-rich cloud processing.
  • 1) Data Selection and Pre-processing:: Feature extraction and selection determine which information is processed locally and which is transmitted to the cloud.Heuristics or lightweight models can guide this decision.
  • 1) Data Selection and Pre-processing:: Data compression reduces edge-cloud bandwidth through quantization, dimensionality reduction, image compression, and intermediate aggregation.Asynchronous aggregation and cluster pairing reduce communication overhead and accelerate convergence.
  • 2) Model Choice on Edge and Cloud:: Model placement trades performance against resources according to device capacity, task complexity, and real-time requirements.
  • 2) Model Choice on Edge and Cloud:: Edge-appropriate models prioritize low computational cost and real-time inference, whereas cloud-appropriate models provide broader analysis and higher accuracy with greater resource demands.
  • 3) Compression Techniques:: Compression combines quantization, pruning, distillation, neural architecture search, adaptive rate scaling, and federated dropout to reduce model size and computation.Quantization can reduce weights and activations from 32-bit floating point to 16-bit or 8-bit formats.
  • 4) Balancing Model Compression and Accuracy: Compression requires balancing efficiency and accuracy because aggressive reduction can degrade performance.Incremental compression, layer sensitivity analysis, hybrid methods, dynamic compression, and domain-specific thresholds help manage this trade-off.
  • 4) Balancing Model Compression and Accuracy: Domain-specific accuracy thresholds align compression strategies with application requirements and deployment environments.

3) VLMs Implementation Across Distribution Locations:

Distributed edge VLMs divide computation, updates, and evaluation between local devices and cloud systems while using federated learning to preserve data locality. Continuous evaluation and adaptation support heterogeneous, evolving deployments, although current VLMs remain resource-intensive and privacy-sensitive.

  • Federated learning trains shared models across decentralized devices without exchanging raw data, supporting privacy-sensitive and heterogeneous applications.Model updates are aggregated into a global model while local data remains stored on participating devices.
  • Model partitioning assigns lightweight inference to edge child models and computation-heavy aggregation or updates to cloud parent models.This hierarchy enables online local inference with minimal on-device memory.
  • Local data policies limit cloud interaction through selective uploading, synchronized model updates, and federated averaging.
  • Differential privacy, secure aggregation, and homomorphic encryption protect data or model updates during distributed training and computation.
  • Federated deployment integrates these privacy techniques to enhance data privacy and security in decentralized VLM networks.
  • Local and global evaluation tracks performance across heterogeneous devices and contexts while identifying when cloud updates or further training are needed.Feedback from difficult local scenarios can improve the global model.
  • Continuous learning, lightweight synchronization, adaptive device updates, and scalable orchestration support evolving edge deployments.Federated dropout and differential synchronization reduce exchanged parameters or data volume.
  • Current VLMs remain difficult to deploy because of high resource requirements, latency, power inefficiency, diverse modalities, and centralized data dependence.The survey recommends lightweight architectures, compression, neural architecture search, and federated learning for real-time, adaptive edge systems.

IV. RECENT ADVANCES

Recent advances extend lightweight VLMs across medical and environmental applications, emphasizing real-time operation on resource-constrained devices. Environmental methods include efficient adaptation, satellite and aerial analysis, UAV navigation, and multimodal change detection.

  • Medical Applications: Lightweight VLMs support medical applications including visual question answering, radiology report generation, Alzheimer’s diagnosis, and ultrasound-based COVID-19 classification.ViLMedic targets efficient edge deployment, while MedBLIP and COVID-LWNet address resource-constrained diagnostic settings.
  • Environmental Monitoring: Environmental monitoring uses VLMs for aerial imaging, environmental change detection, disaster assessment, land-use analysis, and deforestation monitoring.These applications emphasize real-time monitoring in remote or resource-constrained areas.
  • Environmental Monitoring: SATIN achieves high transfer performance in zero-shot classification for environmental surveys on resource-constrained devices.The model is presented as a multi-task metadataset for satellite and aerial imagery classification.
  • UAV Navigation: AerialVLN enables UAVs to navigate complex environments during environmental surveys and wildlife tracking with real-time decision-making.Its lightweight architecture supports deployment on UAV platforms.
  • Remote Sensing Change Detection: ChangeCLIP combines bitemporal visual features, differential feature compensation, and vision-language decoding for remote sensing change detection.It reports state-of-the-art results on five benchmarks: LEVIR-CD (85.20%), LEVIR-CD+ (75.63%), WHUCD (90.15%), CDD (95.87%), and SYSU-CD (71.41%).

C. Autonomous Applications of VLMs

VLMs are applied to autonomous driving, UAV navigation, disaster response, surveillance, healthcare, environmental monitoring, and retail. These applications use multimodal understanding for real-time detection, decision-making, anomaly analysis, and on-device services, while deployment remains constrained by edge resources and modality-alignment challenges.

  • Autonomous Applications: Autonomous-system applications include driving-scene understanding, natural-language command interpretation, traffic-object detection, UAV navigation, and disaster response.These systems target object tracking, safety, search and rescue, and real-time decisions in dynamic environments.
  • Surveillance: Lightweight VLMs support surveillance through anomaly detection, facial recognition, scene analysis, threat detection, crowd monitoring, and unusual-event prediction.Applications span urban surveillance, public spaces, aerial imagery, and sensor-connected smart cities.
  • Cross-Domain Applications: Edge VLMs also serve healthcare, environmental monitoring, precision agriculture, retail, and ecommerce through local diagnostics, imagery analysis, visual search, recommendations, and inventory management.These uses analyze medical, satellite, drone, and product imagery for domain-specific services.
  • Deployment Challenges: The surveyed challenges include visual-textual alignment gaps, expensive large-scale datasets, computational demands, energy efficiency, scalability, hardware miniaturization, and usability.Prior surveys differ in emphasis but collectively address deployment and generalization constraints for edge AI.
  • Deployment Challenges: Compression methods such as knowledge distillation and quantization reduce model size and computation, but specialized compression may be needed for medical imaging and drone-based remote sensing.EdgeVL is reported to reduce model size by up to 93 and improve accuracy by 15% on edge devices.

B. Visual-Language Models Optimization for Edge Networks

Edge VLM optimization combines compact architectures, dynamic resource adaptation, federated learning, parameter-efficient training, multimodal fusion, and task-specific modularity. These strategies target heterogeneous hardware, limited communication and computation, privacy requirements, and real-time performance.

  • Model Architecture: Architecture optimization uses smaller language models, efficient visual encoders, lightweight task-specific components, quantized or pruned language models, and modular adapters.LiteViLA activates lightweight adapters for subtasks such as object detection and scene understanding, while drone systems combine YOLOv7 with VLMs.
  • Runtime Optimization: Dynamic inference and model scaling adjust model complexity to available resources but face unpredictable latency, hardware heterogeneity, and maintenance overhead across model variants.Modular architectures, resource-aware scheduling, automated compression, and device-specific models are proposed responses.
  • Federated Learning: Federated learning supports distributed VLM implementation by fine-tuning models locally, preserving privacy, reducing latency, and adapting to varied hardware without centralized data collection.The approach is especially relevant to healthcare and security applications with sensitive data.
  • Federated Learning: Federated prompt learning, personalized experts, LoRA, gradient compression, selective updates, client sampling, and multimodal collaboration reduce training or communication costs.These methods address resource limits, client heterogeneity, large model updates, and unpaired or unimodal client data.
  • Federated Learning: FL-VLM integration remains limited by data heterogeneity, model divergence, edge computation constraints, and privacy risks, requiring robust aggregation, personalization, lightweight architectures, and secure methods.Differential privacy and secure aggregation are identified as protections for data and updates.
  • Multimodal Adaptation: Cross-modality adaptation targets thermal, depth, and hyperspectral sensors, while lightweight early or late fusion addresses synchronization, latency, and computational overhead.Temporal alignment is difficult when modalities have different data rates and latencies.

F. Security

Edge VLM security concerns span model attacks, insecure transmission, hardware threats, federated-learning poisoning, privacy leakage, and communication overhead. The survey presents defenses including adversarial training, encryption, trusted execution, secure aggregation, and privacy-preserving distributed learning, while identifying interoperability and efficient deployment as open challenges.

  • Security Threats: Edge VLMs are vulnerable to model inversion and adversarial attacks because shared cross-modal representations can expose training information.Adversarial training, gradient masking, randomized transformations, and layered defenses are described as countermeasures.
  • Security Mechanisms: Secure transmission and hardware protection use encryption, dynamic key exchange, secure multiparty computation, trusted execution environments, and protections against tampering or malware.These mechanisms aim to protect distributed data and edge hardware with limited overhead.
  • Federated Security: Federated learning faces poisoning and malicious-update risks, addressed through Byzantine-resilient systems, anomaly detection, differential privacy, homomorphic encryption, and blockchain-based safeguards.These methods target malicious inputs, secure aggregation, computation privacy, and tamper resistance.
  • Privacy: Privacy-aware VLMs keep data local through federated learning, but membership inference, data ownership, collusion, and regulatory compliance remain concerns.Proposed protections include encrypted updates, decentralized client data, differential privacy, and homomorphic encryption.
  • Deployment Constraints: Distributed VLM training and inference must manage communication overhead, energy consumption, latency, and hardware diversity across edge-cloud systems.DeViT, pipeline parallelism, and hardware-aware neural architecture search are cited as approaches for collaborative inference and adaptation.
  • Conclusion: The survey concludes that pruning, quantization, knowledge distillation, and efficient hardware can reduce edge limitations while supporting autonomous systems, surveillance, and medical diagnostics.Interoperability for massive edge deployment remains an open research problem.
Loading 2502.07855v2…