PaperlayerView exact revision on arXiv

Source-linked AI summary

SIMAC: A Semantic-Driven Integrated Multimodal Sensing And Communication Framework

Yubo Peng, Luping Xiang, Kun Yang, Feibo Jiang, Kezhi Wang, Dapeng Oliver Wu

arXiv:2503.08726v1cs.LGcs.AIeess.SP

TL;DR

Single-modality sensing and decoupled sensing-communication systems limit sensing capability, increase communication burden, and constrain service diversity. SIMAC addresses these issues by fusing radar and image semantics, performing channel-adaptive semantic encoding, and using task-specific decoding; simulations report higher sensing accuracy and diversified services.

  • Problem

    Single-modality sensing has limited accuracy and capability, while decoupled sensing and communication increases latency and overhead and single-task systems do not meet diverse sensing demands.

  • Method

    SIMAC fuses radar and image semantics with MSF, maps multimodal semantics and communication parameters into a unified space for LSE encoding, and uses SSD task-specific heads with multi-task learning.

  • Results

    Simulations across four sensing tasks and benchmarks demonstrate higher sensing accuracy and diversified sensing services.

  • Takeaways & Limitations

    SIMAC provides integrated multimodal sensing and communication with parallel sensing services and semantic transmission designed to reduce communication costs.

Abstract

from arXiv · show

Traditional single-modality sensing faces limitations in accuracy and capability, and its decoupled implementation with communication systems increases latency in bandwidth-constrained environments. Additionally, single-task-oriented sensing systems fail to address users' diverse demands. To overcome these challenges, we propose a semantic-driven integrated multimodal sensing and communication (SIMAC) framework. This framework leverages a joint source-channel coding architecture to achieve simultaneous sensing decoding and transmission of sensing results. Specifically, SIMAC first introduces a multimodal semantic fusion (MSF) network, which employs two extractors to extract semantic information from radar signals and images, respectively. MSF then applies cross-attention mechanisms to fuse these unimodal features and generate multimodal semantic representations. Secondly, we present a large language model (LLM)-based semantic encoder (LSE), where relevant communication parameters and multimodal semantics are mapped into a unified latent space and input to the LLM, enabling channel-adaptive semantic encoding. Thirdly, a task-oriented sensing semantic decoder (SSD) is proposed, in which different decoded heads are designed according to the specific needs of tasks. Simultaneously, a multi-task learning strategy is introduced to train the SIMAC framework, achieving diverse sensing services. Finally, experimental simulations demonstrate that the proposed framework achieves diverse sensing services and higher accuracy.

I. INTRODUCTION

Single-modality sensing cannot provide both rich visual information and precise physical measurements, while decoupled sensing and communication increases latency and overhead. SIMAC addresses these gaps by combining multimodal semantic fusion, semantic communication, and task-oriented decoding for diversified sensing services.

  • I. INTRODUCTION: Radar provides physical location and movement information, whereas visual sensing supplies detailed object appearance but is vulnerable to lighting and occlusion.Integrating both modalities is presented as a way to combine complementary sensing strengths.
  • I. INTRODUCTION: Decoupled sensing and communication completes sensing before transmitting results, increasing latency and communication overhead for multimodal systems.These limitations hinder deployment in settings with strict latency and bandwidth constraints.
  • I. INTRODUCTION: Semantic communication conveys message meaning rather than all bits, reducing redundancy and irrelevant data to improve transmission efficiency.The paper positions semantic communication as a basis for addressing communication-cost challenges.
  • I. INTRODUCTION: SIMAC introduces MSF, which extracts unimodal semantics from radar signals and images and fuses them through cross-attention into a multimodal representation.The contribution uses ViT-based visual extraction and complex CNN-based signal extraction.
  • I. INTRODUCTION: SIMAC combines an LLM-based semantic encoder with a multi-task sensing decoder to adapt encoding to communication conditions and support diversified sensing services.The LSE maps multimodal semantics and communication parameters into a unified latent space, while SSD uses task-specific decoding heads.

C. Integrated Sensing and Communication

The system model integrates radar and camera sensing with simultaneous communication, while semantic processing fuses their complementary information before transmission. Its sensing pipeline extracts modality-specific features and produces a unified multimodal semantic representation for integrated processing.

  • C. Integrated Sensing and Communication: Traditional ISAC transmits sensing signals alongside communication data, creating communication overhead and a trade-off between sensing and communication performance.The paper states that this approach is also susceptible to channel noise.
  • C. Integrated Sensing and Communication: SIMAC embeds sensing information into communication semantics and transmits only essential semantic content to reduce communication overhead and improve robustness against channel noise.This semantic-driven integration is contrasted with raw sensing-signal transmission in traditional ISAC systems.
  • C. Integrated Sensing and Communication: The BS transmits an ST’s image and motion parameters while radar and camera sensing and sensing decoding proceed in parallel with communication.The system comprises sensing targets, a BS transmitter, and a user receiver.
  • 1) Sensing Data Acquisition:: The radar model represents each sensing target through angle, distance, and radial velocity and captures its echo using an LFM waveform.The stated motion parameters are θ_n, d_n, and v_n.
  • 2) Semantic Extraction:: Separate semantic extractors process radar echoes and camera images because the modalities have distinct data dimensions and characteristics.The resulting semantic features are then supplied to a semantic fusion module.
  • 2) Semantic Extraction:: A semantic fusion module combines the signal and visual semantic features to generate a comprehensive multimodal semantic representation.The module is described as capturing key information and latent relationships between the two modality-specific features.

3) Semantic Encoding:

SIMAC jointly encodes multimodal sensing semantics and transmits them over a wireless channel while supporting task-specific sensing decoding. Its formulation accounts for semantic distortion, sensing accuracy, and execution latency under multiple sensing tasks.

  • Semantic Encoding:: The JSCC encoder maps fused multimodal semantics into semantic encodings, which are modulated into complex-valued symbols for wireless transmission.QPSK and 16QAM are cited as example modulation techniques.
  • Wireless Communication:: The wireless channel transmits complex-valued symbols under a model including channel gain and additive white Gaussian noise.The channel model is implemented with neural-network approaches to remain compatible with end-to-end backpropagation.
  • Semantic Decoding:: The received semantic encoding is decoded into task-specific outputs including image reconstruction, distance, velocity, and angle estimates.A semantic decoder produces sensing results tailored to multiple tasks.
  • Multi-Task Learning:: SIMAC considers distance, velocity, angle prediction, and image reconstruction as simultaneous sensing tasks with corresponding task losses.The formulation explicitly includes multiple sensing objectives.
  • Optimization Objective:: The framework minimizes semantic distortion and maximizes decoded sensing accuracy while accounting for transmission latency.The total execution time includes semantic extraction, encoding, communication, and decoding computation.
  • Optimization Objective:: The optimization addresses modality heterogeneity, communication-adaptation difficulty, and the challenge of jointly optimizing multiple sensing objectives.These are identified as three key issues in solving the SIMAC optimization problem.

IV. SEMANTIC-DRIVEN INTEGRATED MULTIMODAL SENSING AND COMMUNICATION

SIMAC integrates multimodal sensing with semantic communication through multimodal fusion, channel-adaptive encoding, and task-oriented decoding. The framework is trained for diverse sensing services and transmits essential semantic content rather than complete sensing results.

  • Framework Overview: SIMAC integrates multimodal sensing and semantic communication to address insufficient single-modal information and latency from decoupled sensing and communication.The framework is motivated by real-time service requirements and multimodal sensing limitations.
  • Framework Overview: MSF uses separate signal and visual extractors followed by cross-attention fusion to produce a multimodal semantic representation.Complex CNNs process signals and a ViT-based extractor processes images.
  • Framework Overview: LSE incorporates communication parameters such as SNR and modulation into semantic encoding so representations can adapt to changing communication environments.Communication parameters are expressed as natural-language inputs to the encoder.
  • Framework Overview: SSD uses multiple output heads for distance, velocity, angle, and image-reconstruction tasks.The heads generate task-specific sensing results from received semantic encodings.
  • Framework Overview: Multi-task learning trains the framework to execute different sensing tasks, while semantic integration transmits essential sensing content and is described as reducing communication overhead and improving noise robustness.The stated workflow jointly trains the framework modules.

B. Multimodal Semantic Fusion Module

The MSF module extracts semantic features separately from radar signals and images, then reduces and projects them into compatible representations for multimodal fusion. Its signal branch uses complex convolutions, while its vision branch uses a lightweight BiFormer-based ViT.

  • Multimodal Semantic Fusion Module: MSF processes radar signals and images through separate semantic extractors before fusion.The module combines signal processing with transformer-based image feature extraction.
  • Signal Semantic Extractor: The signal extractor applies complex-valued convolutions to real and imaginary components, followed by complex max-pooling and feature concatenation.Three convolutional layers are each followed by pooling before the components are concatenated.
  • Signal Semantic Extractor: A fully connected layer maps concatenated signal features to a reduced semantic dimensionality.The resulting signal semantic has sequence length Ls and feature dimensionality d.
  • Vision Semantic Extractor: The vision extractor uses a lightweight BiFormer backbone with bilevel routing attention to obtain visual features from images.BiFormer is selected to support extraction accuracy and inference velocity.
  • Vision Semantic Extractor: The vision branch divides images into non-overlapping regions and forms query, key, and value tensors for region-level semantic routing.Region queries and keys determine semantic relationships between image regions.
  • Vision Semantic Extractor: Top-h routing retains the most semantically relevant regions for each region before fine-grained token attention gathers the corresponding keys and values.The routing index matrix stores the indices of the h most relevant regions.
  • Vision Semantic Extractor: A linear projection maps vision features to the same shape as the signal semantic representation.This produces a compatible vision semantic for subsequent fusion.

3) Cross-Attention Fusion Module:

The cross-attention fusion module combines radar and visual semantics through bidirectional attention. Radar and image features alternately serve as queries and key-value inputs, and the resulting outputs are summed and refined.

  • Cross-Attention Fusion Module: Cross-attention fusion combines signal and vision semantics through a bidirectional attention mechanism.Each modality participates in attention-based interaction with the other.
  • Cross-Attention Fusion Module: Signal semantics act as queries with vision semantics as keys and values, and the roles are then reversed.This reciprocal arrangement enables attention in both modality directions.
  • Cross-Attention Fusion Module: The two attention outputs are summed to form the fused multimodal representation.The fusion uses separate learned weights for radar signals and image features.
  • Cross-Attention Fusion Module: Normalization and residual connections refine the fused representation to avoid vanishing gradients.These operations are applied after the attention outputs are combined.
  • Cross-Attention Fusion Module: The resulting MSF design combines complex operations, ViT-based extraction, and attention-driven fusion to integrate latent information from radar signals and images.The paper presents this combination as the module’s overall advantage.
  • Cross-Attention Fusion Module: MSF inference extracts signal and vision semantics, fuses them through cross-attention, and outputs the multimodal semantic representation.The procedure is summarized as a three-step inference algorithm.

C. LLM-Based Semantic Encoder Module

The LLM-based semantic encoder combines multimodal semantics with textual communication parameters, processes them through GPT-2, and produces channel-adaptive encoding. Pooling and nonlinear activation yield an enriched representation for downstream semantic reconstruction and communication.

  • LSE combines MSF-derived multimodal semantic representation with textual communication parameters as its two inputs.The textual parameters are tokenized, embedded, and concatenated with multimodal features along the sequence dimension.
  • GPT-2 processes the fused input and attention mask to generate contextually enriched semantic representations.The fused attention mask is constructed before the representation is generated from GPT-2's final hidden state.
  • Max-pooling reduces the sequence dimension by half before hyperbolic-tangent activation produces the semantic encoding e_n.The inference procedure applies pooling and then activation to obtain e_n.
  • The resulting encoding captures multimodal contextual dependencies and textual semantics for semantic reconstruction and communication.The LSE is intended to bridge textual and non-textual modalities through a comprehensive representation.

D. Sensing Semantic Decoder Module

The sensing semantic decoder reshapes the received encoding into shared spatial features and uses task-specific heads for image reconstruction, distance, angle, and velocity estimation. A ViT decoder reconstructs images while auxiliary heads decode the remaining sensing quantities.

  • SSD converts the received semantic encoding into spatial features that are shared by four task-specific decoding heads.The four tasks are image reconstruction, distance prediction, angle estimation, and velocity estimation.
  • A ViT decoder performs image reconstruction through upsampling, positional embeddings, transformer feature aggregation, and pixel-space prediction.The positional embedding preserves spatial information during transformer processing.
  • Specialized output heads extract angle, velocity, and distance estimates from the shared backbone features.Each head uses its own weights and bias for the corresponding sensing quantity.
  • SSD provides diversified sensing results from the same semantic information, while users can deploy selected output heads locally.The decoder jointly integrates image reconstruction and auxiliary sensing tasks under a unified multitask objective.

E. Multi-Task Learning-Based Training Process

SIMAC uses multi-task learning to jointly optimize image reconstruction and sensing predictions under dynamically generated communication conditions. Training computes task-specific losses, combines them, and updates the framework parameters through backpropagation.

  • Multi-task learning jointly trains image reconstruction, angle estimation, and distance prediction within one framework.The stated objective is to support diversified sensing services through simultaneous optimization of multiple tasks.
  • Image reconstruction uses L1 loss, while angle, velocity, and distance prediction use mean squared error loss.The total objective is a weighted sum of the task losses.
  • The training loop generates communication parameters from dynamic SNR and modulation settings before predicting the sensing outputs.The predictions include reconstructed images, distances, velocities, and angles.
  • The total loss is backpropagated to update the framework parameters with an optimizer during each training batch.Algorithm 5 repeats this process across epochs and batches from the training dataset.

V. EXPERIMENTAL SETUP AND NUMERICAL RESULTS

The experiments construct a car-focused multimodal dataset from VIRAT videos, derive sensing labels from image detections and tracking, and evaluate communication under an AWGN channel. The setup varies SNR and modulation while using radar and image inputs for sensing tasks.

  • The dataset is built from three VIRAT scenes sampled at one frame per second, yielding approximately 10,000 RGB images.Cars are detected with YOLOv10 and isolated from frames using the segment anything model.
  • Distance and angle labels are computed relative to a base station assumed to occupy the lower-right image corner.The base-station location and detected bounding-box centers determine the geometric labels.
  • Object velocity is estimated from center-point displacement across consecutive tracked frames divided by the frame duration.YOLO tracking is used to associate the same object across frames.
  • The communication simulation uses an AWGN channel with bandwidth B = 1 kHz, power P = 1 W, and SNR varied from 0 dB to 25 dB.Modulation changes across SNR ranges from BPSK through QPSK, 8PSK, and 16QAM.
  • The radar configuration uses fc = 10 GHz, Fs = 60 MHz, and K = 10 antennas, with car target RCS set to ρ_n = 100.Inference evaluates fixed SNR conditions at 0, 10, 15, 20, and 25 dB.

3) Benchmark Schemes:

SIMAC is evaluated against ablated variants and DeepJSCC using task-specific sensing metrics and visualized outputs. The framework produces multimodal sensing outputs, reconstructs images, remains resilient to channel noise, and achieves higher sensing accuracy across three scenes.

  • Benchmark schemes: The benchmark set includes SIMAC variants without the LSE or SSD modules, plus DeepJSCC for image reconstruction evaluation.SIMAC (w/o LSE) excludes communication parameters, while SIMAC (w/o SSD) removes multiple output heads and trains each sensing task independently.
  • Evaluation metrics: RMSE evaluates distance, velocity, and angle prediction using normalized predicted values and ground truth.RMSE measures the absolute average deviation between predictions and ground truth.
  • Evaluation metrics: PSNR and SSIM evaluate reconstructed-image quality, with PSNR reported in decibels and SSIM accounting for luminance, contrast, and structure.PSNR uses higher values to indicate better reconstruction quality.
  • Visualization of sensing results: SIMAC processes images and radar signals to produce reconstructed images and predicted motion attributes, while supporting diversified outputs for one image.The framework’s reconstructed images restore missing parts and exhibit superior visual quality in the reported comparisons.
  • Visualization of sensing results: SIMAC demonstrates resilience to channel noise and higher sensing accuracy across three scenes than SIMAC (w/o SSD) and DeepJSCC.The reported comparison attributes the lower accuracy of SIMAC (w/o SSD) and DeepJSCC to lacking signal-modality assistance for precise visual positioning.
  • Visualization of sensing results: The MSF module uses radar semantic information as queries to localize corresponding spatiotemporal image features during training.The reported findings connect multimodal fusion with target localization and motion-attribute estimation.

2) Evaluation for Distance Sensing:

SIMAC achieves the best sensing accuracy across distance, velocity, and angle prediction, while also maintaining strong image-reconstruction quality across channel conditions. Ablations show that both LSE and SSD contribute materially to performance.

  • Distance sensing: SIMAC achieves the lowest distance-prediction RMSE across all SNR levels and scenes.SIMAC (w/o LSE) degrades as noise increases, while SIMAC (w/o SSD) has the worst RMSE.
  • Velocity sensing: SIMAC achieves the best velocity-prediction RMSE across all SNR levels and scenes.Removing LSE yields RMSE around 0.2, whereas removing SSD yields approximately 0.8.
  • Angle sensing: SIMAC achieves the best angle-sensing RMSE across all SNR levels and scenes.SIMAC (w/o LSE) remains around 0.2, while SIMAC (w/o SSD) remains around 0.3.
  • Image reconstruction: SIMAC exceeds 24 dB PSNR and 0.85 SSIM in Scene 1, outperforming the ablations and DeepJSCC.DeepJSCC remains below 18 dB PSNR and below 0.7 SSIM, while SIMAC without LSE or SSD performs lower than the complete framework.
  • Overall evaluation: The complete SIMAC framework maintains high reconstruction quality, with LSE and SSD both contributing to PSNR and SSIM performance.The reported comparisons cover varying SNR conditions and multiple scenes.
Loading 2503.08726v1…