PaperlayerView exact revision on arXiv

Source-linked AI summary

EMMI: Edge Multi-Modal Intelligence for Communication-Efficient MLLM Inference via Fused Representation Compression

Motahare Mounesan, Irfan Khan

arXiv:2609.11058v1cs.LGcs.DC

TL;DR

Resource-constrained edge platforms struggle to support MLLMs because multimodal reasoning requires substantial computation, memory, and communication. EMMI fuses and compresses multimodal representations at the edge for server-side reasoning, reducing communication payload by 32× while maintaining comparable accuracy and cutting estimated latency by up to 3.4× under bandwidth constraints.

  • Problem

    MLLM deployment on resource-constrained edge platforms is limited by substantial computational, memory, and communication demands.

  • Method

    EMMI performs modality-specific encoding, cross-modal fusion, and learned compression at the edge, transmitting compact representations for server-side MLLM reasoning.

  • Results

    32× payload reduction preserves comparable downstream accuracy, while estimated end-to-end latency falls from approximately 903ms to 269ms, a 3.4× improvement under 0.1Mbps bandwidth.

  • Takeaways & Limitations

    EMMI shifts inference away from communication-bound operation while preserving information required for downstream reasoning, with benefits strongest when communication is the bottleneck.

Abstract

from arXiv · show

Recent advances in multimodal large language mod- els (MLLMs) have opened new opportunities for edge intelligence by enabling reasoning across heterogeneous sensor modalities, such as vision, text, and telemetry data. However, deploying these capabilities on resource-constrained edge platforms remains challenging due to the substantial computational, memory, and communication demands of modern MLLMs. Rather than transmitting raw sensor observations or partitioning neural networks at intermediate layers, Edge Multi-Modal Intelligence (EMMI) communicates a compact representation between edge devices and server resources, enabling communication-efficient edge MLLM inference. To achieve this, EMMI performs modality-specific encoding, cross-modal representation fusion, and learned compression at the edge, transmitting only a compact latent representation to server-side resources for high-capacity MLLM reasoning. This representation-centric design reduces communication overhead, preserves local data privacy, and provides a fixed-size interface between heterogeneous edge devices and server-side MLLMs. Evaluation on a representative multimodal benchmark demonstrates that EMMI can reduce the communication payload by 32x while maintaining comparable downstream accuracy, resulting in up to a 3.4x reduction in estimated end-to-end inference latency under bandwidth-constrained edge conditions.

I. INTRODUCTION

MLLMs offer flexible cross-modal reasoning, but their computational, memory, and communication demands hinder deployment on resource-constrained edge platforms. EMMI addresses this gap by fusing and compressing multimodal representations at the edge before server-side reasoning.

  • I. INTRODUCTION: MLLMs capture higher-order relationships across heterogeneous inputs, but their computational, memory, and communication requirements exceed many edge platforms’ resources.
  • I. INTRODUCTION: Existing strategies reduce model complexity, offload computation, or transmit intermediate representations, but can sacrifice expressive capability or add communication overhead and latency.
  • I. INTRODUCTION: EMMI performs modality-specific encoding, cross-modal fusion, and compression at the edge before transmitting a unified representation for server-side reasoning.This changes the communication boundary from raw inputs or intermediate activations to a compact representation.
  • I. INTRODUCTION: EMMI provides a fixed-size interface between heterogeneous edge devices and server-side reasoning models while preserving task-relevant multimodal information.

II. RELATED WORK

Traditional multimodal systems often use lightweight, task-specific models, whereas MLLMs support general-purpose reasoning across heterogeneous inputs. Their computational and memory requirements nevertheless limit deployment on resource-constrained edge platforms.

  • II. RELATED WORK: MLLMs extend multimodal intelligence toward general-purpose reasoning across heterogeneous inputs, beyond task-specific lightweight architectures.
  • II. RELATED WORK: Computational and memory requirements remain major barriers to deploying MLLMs on resource-constrained edge platforms.

III. SYSTEM MODEL AND SOLUTION APPROACH

EMMI uses an edge–server pipeline that encodes, fuses, and compresses multimodal information at the edge, then decompresses it for server-side reasoning. This representation-centric boundary avoids transmitting raw sensor streams or architecture-dependent activations.

  • III. SYSTEM MODEL AND SOLUTION APPROACH: EMMI encodes, fuses, and compresses multimodal information at the edge before server-side decompression and reasoning.

A. System Model

The system distributes multimodal inference across resource-constrained edge devices and high-capacity servers connected by bandwidth-limited wireless links. Its design balances edge and server computation while minimizing communication overhead.

  • A. System Model: The system connects resource-constrained edge platforms to remote servers with high-capacity MLLMs through bandwidth-limited wireless links.
  • A. System Model: End-to-end inference latency comprises edge processing, communication, and server-side reasoning.
  • A. System Model: Communication latency depends on transmitted data size S_tx and available bandwidth B.
  • A. System Model: The system must balance computational workload across edge and server resources while minimizing communication overhead under edge constraints.

B. EMMI Framework

EMMI separates edge-side multimodal representation generation from server-side MLLM reasoning by fusing and compressing inputs before transmission. The server reconstructs the representation for downstream reasoning without requiring raw observations or modality-specific intermediate features.

  • Edge–Server Architecture: EMMI generates compact multimodal representations at the edge instead of transmitting raw sensor streams or architecture-dependent activations.The edge pipeline performs modality-specific processing, fusion, and compression before communication.
  • Edge–Server Architecture: Cross-modal fusion combines modality-specific representations before communication to preserve relationships relevant to downstream MLLM reasoning.The fused representation captures interactions among heterogeneous inputs rather than transmitting independent modality features.
  • Edge–Server Architecture: Representation compression reduces dimensionality while preserving information relevant to downstream reasoning and limiting exposure of raw sensor observations.The resulting compact representation is transmitted to the server for reconstruction and inference.
  • Server-side MLLM Reasoning: The server decompresses the transmitted representation, projects it into the MLLM embedding space, and applies task-specific reasoning.This pipeline produces the final task-specific inference result without access to original sensor observations.
  • System-level Benefits: Decompression decouples the compression mechanism from downstream reasoning, allowing compression techniques or ratios to change without retraining the reasoning module.Reconstructed representations can also be batched for parallel server-side MLLM processing.

C. Proposed Method

EMMI fuses modalities through explicit pairwise interactions and supports task-agnostic or task-aware compression objectives. Both strategies share the same compression architecture while differing in how the latent representation is trained.

  • Cross-Modal Fusion: Pairwise interaction-based fusion combines modality features with concatenation, absolute differences, and element-wise products to capture complementary and correlated characteristics.For more than two modalities, pairwise features are aggregated into a unified multimodal representation.
  • Representation Compression: Both compression strategies operate on the same fused representation and share the same edge-side architecture, differing only in their training objectives.This preserves a common lightweight compression and transmission pipeline across training settings.
  • Representation Compression: Task-agnostic compression jointly optimizes reconstruction fidelity and cross-modal alignment to learn a reusable latent representation without downstream task labels.The reconstruction and contrastive objectives preserve multimodal information and cross-modal structure.
  • Representation Compression: Task-aware compression uses downstream labels in supervised contrastive learning to optimize the compressed representation for a target inference objective.The reconstruction term preserves fused-representation information while the supervised objective separates samples according to task labels.

IV. EVALUATION

The evaluation first tests whether compressed representations preserve multimodal task information, then measures their communication and inference efficiency under constrained network conditions.

  • Evaluation Goals: The evaluation compares task performance across compressed representations and measures communication overhead and end-to-end latency under constrained network conditions.It examines both representation quality and system-level efficiency.
  • Evaluation Goals: The study evaluates compression strategies by first examining preservation of multimodal information and then assessing communication and inference efficiency.This ordering connects representation quality with downstream system impact.

A. Experimental Setup

Experiments use MS-COCO image-caption matching with full-size and lightweight vision-language encoders, a fixed 64-dimensional compression target, and a frozen LLaVA server backbone. Accuracy, compression, and latency are measured across controlled hardware settings.

  • Dataset and Multimodal Encoders: Experiments use MS-COCO image-caption pairs for image-text matching with CLIP ViT-B/32 and MobileCLIP2-S0 encoders.The encoders produce 512-dimensional embeddings, while the image encoders contain 87M and 11.4M parameters, respectively.
  • Compression Configuration: All compression approaches target a 64-dimensional latent representation for comparison.The setup evaluates compression effectiveness at a common representation size.
  • Server-side Inference: Server-side inference uses a frozen 6.6B-parameter LLaVA-1.5-7B backbone and a trainable two-layer MLP mapping received representations to eight soft-prompt tokens.A binary classification head processes the backbone’s final hidden representation.
  • Evaluation Environment: Accuracy and server-side latency are measured on an NVIDIA Quadro RTX 6000, while edge computation is evaluated using one CPU thread on an x86-64 compute node.The separate setup provides controlled comparisons and a reproducible proxy for resource-constrained edge execution.

B. Results and Discussion

EMMI’s learned, task-aware compression preserves downstream accuracy while reducing communication payload and latency, with benefits strongest under bandwidth-constrained conditions.

  • Compression Performance: Compression effectiveness depends strongly on representation and objective, with generic methods degrading MobileCLIP accuracy by up to 46.4 percentage points.BlockPCA and LDA remain near the uncompressed baseline, whereas PCA, AE, and VAE substantially reduce accuracy.
  • Compression Performance: 32× payload reduction preserves accuracy within 0.16 percentage points of uncompressed baselines for TaskAwareAE on CLIP and MobileCLIP.TaskAwareAE reaches 98.08% accuracy for CLIP and 98.32% for MobileCLIP.
  • Representation and Communication Efficiency: Estimated end-to-end latency falls from approximately 903ms to 269ms at 0.1Mbps, a 3.4× improvement.Transmission latency decreases from 655.4ms to 20.5ms while compression adds 0.641ms overhead.
  • Representation and Communication Efficiency: Representation processing adds negligible edge-side overhead, with fusion at 0.013ms and all compression methods below 1ms.AE-based compression has the highest reported overhead at 0.641ms.
  • Representation and Communication Efficiency: The 32× payload reduction saves approximately 635ms at 0.1Mbps, but savings decrease to 0.68ms at 100Mbps 5G.Savings are 6.4ms at 10Mbps LTE and 1.26ms at 50Mbps WiFi.

V. CONCLUSION

EMMI enables edge MLLM inference by encoding, fusing, and compressing multimodal inputs before transmitting compact representations for server-side reasoning. It reduces communication overhead while preserving task-relevant information and substantially improving estimated latency under bandwidth constraints.

  • EMMI performs modality-specific encoding, cross-modal fusion, and learned compression at the edge before transmitting compact representations for server-side MLLM reasoning.
  • Shifting communication from raw inputs and intermediate features to compressed representations reduces overhead while preserving task-relevant information.
  • 32× lower communication payload and 3.4× lower estimated end-to-end latency were achieved under bandwidth-constrained IoT conditions while remaining within 0.16 percentage points of uncompressed baselines.
Loading 2609.11058v1…