EMMI: Edge Multi-Modal Intelligence for Communication-Efficient MLLM Inference via Fused Representation Compression

I. INTRODUCTION

• Edge systems increasingly need multimodal reasoning to interpret visual observations, sensor measurements, telemetry, and contextual data for timely, context-aware decisions.MLLMs offer a foundation by processing heterogeneous inputs in a unified architecture.Source: §I · p. 1
• MLLMs enable contextual reasoning across heterogeneous inputs, but their computational, memory, and communication demands exceed the resources available on many edge platforms.Reducing model size can constrain expressiveness, while offloading or split execution can incur communication overhead and latency.Source: §I · p. 1
• EMMI performs modality-specific encoding and cross-modal fusion at the edge, then compresses and transmits a unified representation for server-side reasoning.This shifts the communication boundary away from raw streams and frequent intermediate-feature exchanges, providing a fixed-size interface between heterogeneous edge devices and server-side models.Source: §I · p. 2

Multimodal Intelligence under Edge Constraints

• Compact MLLMs reduce model size and computation, but remain constrained relative to full-capacity MLLMs on resource-limited edge platforms.Traditional multimodal systems use lightweight task-specific architectures, whereas general-purpose MLLMs such as Flamingo, BLIP-2, and LLaVA face computational and memory limits at the edge.Source: §II · p. 2

On-Device MLLM Optimization

• Hardware acceleration, quantization, and efficient MLLM architectures [13]–[15] improve edge efficiency but retain a trade-off with high-capacity multimodal reasoning.EMMI instead preserves high-capacity MLLM reasoning while shifting the communication boundary toward compact multimodal representations generated at the edge.Source: §II · p. 2

Split Computing and Collaborative Inference

• Split computing – and collaborative LLM inference [23]– optimize computation placement, but communication remains tied to model-dependent activations, features, or states.EMMI changes what crosses the edge–server boundary by sending a compressed unified multimodal representation for server-side MLLM reasoning.Source: §II · p. 2

Communication-Efficient Representation Transfer

• Feature compression [27], task-oriented communication,, learned compression, and TOFC [31] reduce transmission overhead, but typically transmit modality-specific representations.EMMI fuses modalities at the edge and sends a unified latent, providing a fixed-size interface between heterogeneous sensors and high-capacity MLLMs while retaining open-ended reasoning flexibility.Source: §II · p. 2

• EMMI introduces an edge–server system model and solution design for communication-efficient MLLM inference under edge constraints.Source: §III · p. 2
• At the edge, EMMI encodes, fuses, and compresses multimodal representations.Source: Figure 1 · p. 3
• At the server, EMMI decompresses the transmitted representation and performs reasoning.Source: Figure 1 · p. 3

A. System Model

• Edge devices collect and locally process heterogeneous sensor observations, while remote servers provide high-capacity MLLM reasoning over them.The system supports one or more edge platforms connected to servers through bandwidth-limited wireless links.Source: A. System Model · p. 3
• Inference is split between edge platforms and servers because high-capacity MLLMs require substantial computation and memory resources.Information from edge observations must be transferred over wireless links for server-side processing.Source: A. System Model · p. 3
• End-to-end inference latency consists of edge processing, communication, and server-side reasoning.The passage denotes these components as Tedge, Tcomm, and Tserver.Source: A. System Model · p. 3
• Communication latency depends on transmitted data size and available bandwidth, so workload allocation must limit communication overhead under edge resource constraints.The system balances computation assigned to edge and server resources while minimizing communication overhead.Source: A. System Model · p. 3

B. EMMI Framework

• EMMI encodes each modality independently, fuses features before communication, and compresses the fused representation into an edge–server interface for server-side reasoning.The fusion captures relationships among heterogeneous inputs, and compression reduces dimensionality while preserving information relevant to downstream reasoning.Source: B. EMMI Framework · pp. 3–4
• At the server, EMMI decompresses the representation, projects it into MLLM embedding space, and applies task-specific reasoning and prediction while leaving edge representation generation unchanged.Inference does not require access to original sensor observations or modality-specific intermediate features.Source: B. EMMI Framework · p. 4
• EMMI keeps raw sensor observations local by transmitting compact multimodal representations instead of raw sensor data over the edge–server link.This reduces exposure of raw observations while preserving information required for server-side reasoning.Source: B. EMMI Framework · p. 4
• Decompressing into the fused representation space decouples compression from downstream reasoning, allowing compression techniques or ratios to change without retraining the server-side reasoning module.Reconstructed representations can also be batched for parallel server-side inference, amortizing MLLM reasoning costs across requests.Source: B. EMMI Framework · p. 4

Cross-Modal Fusion

• EMMI fuses modality-specific representations before compression, creating a unified representation that captures modality information and cross-modal interactions.This reduces the need to transmit separate modality representations to the server.Source: C. Proposed Method · p. 4
• Pairwise interaction-based fusion combines modality features with explicit cross-modal features; for multiple modalities, pairwise features are aggregated into a unified representation.Concatenation retains individual modality information, while element-wise difference and multiplication capture complementary and correlated characteristics.Source: C. Proposed Method · p. 4

Representation Compression

• With no task labels or when reusable representations are desired, task-agnostic compression learns a general-purpose representation; with labels, task-aware compression optimizes it for the target inference objective.Both strategies use the same fused representation and edge-side compression architecture, differing only in their training objectives.Source: C. Proposed Method · pp. 4–5
• Task-agnostic compression jointly minimizes reconstruction error and InfoNCE alignment [32], preserving multimodal information and alignment to individual modality embeddings without task labels.The reconstruction term limits distortion between the fused representation and its reconstruction; the alignment objective preserves cross-modal structure.Source: C. Proposed Method · p. 5
• With downstream labels, task-aware compression replaces task-independent alignment with supervised contrastive learning, shaping the latent space so similar-label samples move closer and different-label samples separate.Task supervision enters only through the contrastive objective during training, preserving the lightweight edge-side compression and transmission pipeline.Source: C. Proposed Method · p. 5

• The evaluation compares task performance across methods to assess how well compressed representations preserve multimodal information.Source: §IV · p. 5
• It also measures communication overhead and end-to-end latency under constrained network conditions.Source: §IV · p. 5

Dataset and Multimodal Encoders; Compression Configuration

• MS-COCO image-text matching compares CLIP ViT-B/32 (87M parameters) and MobileCLIP2-S0 (11.4M), both producing 512-dimensional embeddings compressed to 64 dimensions.MS-COCO image-caption pairs serve as the representative vision-language benchmark.Source: A. Experimental Setup · p. 5

Server-side Inference

• A two-layer trainable MLP maps received representations to eight soft-prompt tokens for frozen LLaVA-1.5-7B inference, followed by a binary classification head.The LLaVA backbone has 6.6B parameters and remains frozen.Source: A. Experimental Setup · p. 5

Baselines

• Compression strategies are compared with uncompressed transfer, PCA, blockwise PCA, LDA, and VAE baselines.BlockPCA applies PCA independently to separate blocks of the fused representation.Source: A. Experimental Setup · p. 5

Training Protocol

• Training uses two decoupled stages: edge compression is trained first, then frozen while server projection and classification heads train on cached latents from a 72K/5K/5K split.Pretrained vision-language encoders and LLaVA remain frozen, and model selection uses validation loss.Source: A. Experimental Setup · p. 5

Evaluation Metrics

• Evaluation measures test accuracy, compression ratio, communication payload in bytes, and end-to-end latency across edge generation, compression, transmission, and server inference.End-to-end latency includes all four listed stages.Source: A. Experimental Setup · p. 5

Evaluation Environment

• Accuracy, compression effectiveness, and server inference latency are measured on an NVIDIA Quadro RTX 6000, while edge overhead is evaluated on a single CPU thread on an x86 64 compute node.The CPU-thread setup provides a reproducible proxy for resource-constrained execution.Source: A. Experimental Setup · p. 6

Compression Performance

• On MobileCLIP, PCA, AE, and VAE lose 41.06, 46.4, and 28.5 percentage points, while BlockPCA and LDA remain near baseline; CLIP tolerates several conventional methods.LDA achieves 97.78%/98.33% and, like TaskAwareAE, substantially outperforms generic methods on MobileCLIP.Source: B. Results and Discussion · pp. 6–7
• 98.08% CLIP and 98.32% MobileCLIP accuracy: TaskAwareAE matches uncompressed baselines within 0.16 percentage points while reducing payload 32×.It uses supervised contrastive learning to optimize compressed representations for downstream tasks.Source: B. Results and Discussion · pp. 6–7
• 94.09% CLIP and 90.30% MobileCLIP accuracy with TaskAgnosticAE support label-free reuse, but label-free BlockPCA reaches 97.12% and 97.85%, respectively.TaskAgnosticAE learns reusable latents without task labels; BlockPCA is a closed-form method requiring neither labels nor training.Source: B. Results and Discussion · p. 6

Representation and Communication Efficiency

• 3.4×: at 0.1Mbps, EMMI cuts estimated end-to-end latency from approximately 903ms to 269ms, reducing transmission latency from 655.4ms to 20.5ms.The 32× payload reduction adds 0.641ms of compression overhead.Source: B. Results and Discussion · p. 7
• Cross-modal fusion requires 0.013ms, while every evaluated compression method takes less than 1ms (AE-based compression is highest at 0.641ms); modality encoding accounts for over 99% of edge-side computation.Source: B. Results and Discussion · p. 7
• At 0.1Mbps, the 32× payload reduction saves approximately 635ms per inference.Savings fall to 6.4ms at 10Mbps LTE, 1.26ms at 50Mbps WiFi, and 0.68ms at 100Mbps 5G; as bandwidth rises, latency benefits diminish and compression primarily saves communication and provides a compact edge–server interface.Source: B. Results and Discussion · p. 7, Table III · p. 7

V. CONCLUSION

• EMMI performs modality-specific encoding, cross-modal fusion, and task-aware learned compression at the edge, sending compact representations to server-side resources for reasoning.This shifts the communication boundary from raw inputs and intermediate features to compressed representations, reducing communication overhead while preserving task-relevant information.Source: §V · p. 7
• Performance stayed within 0.16 percentage points of uncompressed baselines, while the payload fell 32× and estimated end-to-end latency improved 3.4× under bandwidth-constrained IoT conditions.The paper presents these results as evidence for jointly designing multimodal representations and compression for efficient edge MLLM inference.Source: §V · p. 8
• Future work will evaluate EMMI on edge hardware, extend it to more complex modalities and heterogeneous devices, and address non-aligned modality encoders.Source: §V · p. 8