PaperlayerView exact revision on arXiv

Source-linked AI summary

Stacked Intelligent Metasurface-Based MultiModal Feature Fusion for Collaborative Inference in Edge Networks

Yushi Wang, Zhengxin Yu, Guhan Zheng, Haris Pervaiz, Zheyi Chen, Shunqing Zhang

arXiv:2610.13364v1cs.DC

TL;DR

Collaborative inference faces communication and computation overhead, alongside heterogeneous data, models, and channels. SIM-CI uses an edge metasurface to process over-the-air aggregated features, with feature adaptation and channel compensation. It reaches 98.6% of conventional digital-domain CI accuracy while reducing computational energy by 25% and requiring 75% fewer transmission time slots.

  • Problem

    Collaborative inference must handle heterogeneous user features, models, and channels, while digital-domain feature transmission and fusion consume spectrum, computation, latency, and energy.

    Source: A. Challenges · pp. 1–2

  • Method

    SIM-CI aggregates multiuser features over the air for edge-SIM inference, using a tri-heterogeneity adapter and dual channel robustness through CSI-aware adaptation and hybrid beamforming.

    Source: B. Motivation and Contributions · p. 2

  • Results

    SIM-CI reaches 98.6% of conventional digital-domain CI accuracy, reduces computational energy by 25%, and requires 75% fewer transmission time slots.

    Source: §VI · p. 13

  • Takeaways & Limitations

    The results present SIM-CI as a promising pathway for scalable, low-latency edge intelligence in 6G systems.

    Source: §VI · p. 13

Abstract

from arXiv · show

In navigating the challenges of computational and communication overhead within the 6G-era Collaborative Inference (CI), this paper explores the feasibility of employing Stacked Intelligent Metasurface (SIM) to perform CI, termed SIM-CI. The SIM functions as a physical computing unit, receiving multi-user Over-The-Air (OTA) aggregated features within the same spectrum. It thus executes the necessary spatial-frequency combining and inference in a single physical wave propagation.SIM-CI also incorporates a tri-heterogeneity robustness adapter to address the network heterogeneity, such as multi-modal data and user inference models, as well as a quality-aware contrastive approach that align features from different modalities and extraction levels into a unified feature space. Considering the heterogeneous and dynamic channels, we further integrate Channel State Information (CSI) as an additional input into the adapter design, enabling users' early perception of potential channel states. Moreover, to maximally mitigate the impact of channel variations, we design a new hybrid beamforming network that combines the robustness of model-driven pre-equalization with the adaptability of data-driven learning. Experimental results demonstrate the effectiveness of our SIM-CI compared to digital-domain inference, achieving 98.6\% accuracy to conversional digital-domain CI approaches under various heterogeneous conditions, while reducing computational energy by 25\% and requiring 75\% fewer transmission time slots.

I. Introduction

Collaborative inference can fuse heterogeneous users’ features, but transmitting and digitally combining them creates communication and computation overhead. The paper proposes SIM-CI to handle data, model, and channel heterogeneity while combining transmission and inference in the physical domain.

I. Introduction

  • Digital collaborative inference faces spectrum use from feature uploads and added computation latency and energy from edge-side fusion.SIM propagation offers a route to reduce the digital-processing burden, but physical superposition makes heterogeneous information harder to align and filter.Source: §I · p. 1, A. Challenges · p. 1

A. Challenges

  • Data modalities and model depths can differ across users, while dynamic channels distort feature amplitudes and phases before fusion.The paper identifies these as joint challenges for SIM-based inference, where corrupted features may enter fusion without digital filtering.Source: A. Challenges · pp. 1–2

B. Motivation and Contributions

  • SIM-CI pairs a GNN-based tri-heterogeneity adapter with dual channel robustness: CSI-aware implicit sensing and explicit hybrid beamforming compensation.The adapter aligns modality and abstraction-level features; the beamforming network combines model-driven pre-compensation with data-driven adaptation.Source: B. Motivation and Contributions · p. 2
  • SIM-CI combines parallel over-the-air feature transmission with physical-domain inference, using an edge SIM to process the superimposed waveform.The framework places a tri-heterogeneity adapter and beamforming networks before transmission.Source: B. Motivation and Contributions · p. 2

II. Related Works

  • Prior work addresses modality alignment, adaptive inference, beamforming, and SIM processing, but does not jointly align modalities and model depths with channel compensation for multiuser inference.The related-work discussion highlights that digital-domain flexibility is limited when features are directly superimposed and processed by SIM.Source: §II · p. 3

III. System Model

  • The system considers multiple users observing a common scene and collaboratively inferring its object categories at an edge server.Users and server communicate over a multiuser OFDM system.Source: §III · p. 3

A. Data Acquisition and Feature Extraction

Users extract features from heterogeneous observations for transmission to an edge server. The system model describes OFDM wireless communication and contrasts it with conventional edge-side feature fusion.

A. Data Acquisition and Feature Extraction

  • Users process heterogeneous observations, including images or text, with local models and project encoder outputs into a common feature dimension.The projected outputs form visual or textual tokens for subsequent transmission.Source: A. Data Acquisition and Feature… · p. 3

B. Wireless Communication Model

  • Extracted features are channel-encoded and modulated into complex symbol vectors for uplink transmission in a multiuser OFDM system.The channel is frequency-selective, with propagation paths described by complex gain, delay, and angle of arrival.Source: B. Wireless Communication Model · pp. 3–4

C. Conventional Edge Processing

  • Conventional edge processing decodes and demodulates user transmissions, then fuses recovered features with a CI model to produce an inference result.Source: C. Conventional Edge Processing · p. 4

IV. The Proposed SIM-CI Framework

SIM-CI adapts heterogeneous user features and transmissions before their over-the-air aggregation. An edge SIM processes the aggregated signal for classification, with a digital head producing final probabilities.

A. SIM-CI design

  • The SIM comprises propagation layers and trainable meta-atoms, whose optimized physical states process aggregated multiuser features for collaborative inference.Source: A. SIM-CI design · p. 4

1) Data Acquisition and Feature Extraction:

  • The adapter combines user token sequences, feature-quality indicators, and estimated channel vectors into a unified, channel-aware semantic feature.The resulting feature is independent of the original modality or feature depth and is passed to the beamforming network.Source: 1) Data Acquisition and Feature… · p. 4
  • A hybrid beamforming network maps each real-valued semantic feature and channel estimate to a complex OFDM symbol vector, jointly encoding semantics and pre-compensating channel distortion.The channel estimate includes error to represent impairments such as pilot contamination and feedback delays.Source: 1) Data Acquisition and Feature… · pp. 4–5

2) Wireless Communication Model:

  • Users transmit simultaneously in the same band so their signals naturally superimpose, letting the wireless channel act as an analog adder for OTA feature aggregation.Amplitude fading and phase rotation can disrupt feature geometry; transmitter pre-equalization and phase pre-alignment compensate for these effects.Source: 2) Wireless Communication Model · p. 5

3) Novel SIM processing:

  • The edge SIM receives the OTA-aggregated space-frequency signal and maps it to class-output ports, where measured signal intensities yield logits for a digital classification head.The head converts the logits into normalized class probabilities.Source: 3) Novel SIM processing · p. 5

B. Tri-heterogeneity Robustness Adapter based on GNN

The adapter uses a GNN to align heterogeneous token features through a graph of intra- and cross-modal similarities. Quality-aware training and channel-aware inputs support fused representations for beamforming.

B. Tri-heterogeneity Robustness Adapter based on GNN

  • The adapter's GNN builds on raw visual and text features to align tokens across heterogeneous data and model representations.It represents the tokens as nodes in a graph whose similarity structure captures intra-modal and cross-modal relationships.Source: B. Tri-heterogeneity Robustness… · p. 5
  • Intra-modal similarities use a kNN-constrained RBF kernel, while cross-modal similarities combine top matches with row-average context.The resulting similarity matrix assembles visual-visual, visual-text, text-visual, and text-text blocks.Source: B. Tri-heterogeneity Robustness… · pp. 5–6
  • Across GNN layers, graph-structure-biased attention combines learned attention with the precomputed similarities.The hyperparameter λatt balances learned attention against the fixed graph topology.Source: B. Tri-heterogeneity Robustness… · p. 6
  • Pooling produces a global representation for each modality, and their concatenation forms the user feature sent to beamforming.The fused representation has semantic dimension Dsem = 2d.Source: B. Tri-heterogeneity Robustness… · p. 6
  • Quality-aware intra-modal contrastive loss aligns similar neighboring tokens and gives higher-quality features stronger influence.The quality metric is based on feature extraction depth.Source: B. Tri-heterogeneity Robustness… · pp. 6–7
  • Concatenating channel information with the initial features yields a channel-aware representation processed by the same GNN logic.The resulting fused feature is described as robust to channel distortions and adaptive to varying abstraction depths.Source: B. Tri-heterogeneity Robustness… · p. 7

C. Beamforming Network .

The beamforming network maps each user's fused feature to transmission symbols. It combines channel-aware feature inputs and quality-aware contrastive training with a hybrid of model-driven and data-driven mapping.

C. Beamforming Network .

  • Quality-aware cross-modal contrastive loss aligns global visual and text representations from the same sample and separates mismatched pairs.The loss is guided by feature quality; the complete objective is a weighted sum of contrastive terms.Source: C. Beamforming Network · p. 7
  • The adapter incorporates CSI by encoding estimated channel vectors into a compact embedding concatenated with each user's raw features.These channel-aware initial features are processed by the GNN to produce robust representations.Source: C. Beamforming Network · p. 7
  • The beamforming network maps each user's fused feature to a frequency-domain symbol vector for transmission using a hybrid approach.The design combines data-driven and model-driven strategies to balance adaptability and robustness.Source: C. Beamforming Network · p. 7

1) Model-Driven OTA Alignment Symbol Mapping:

The model-driven branch uses an MMSE-inspired pre-equalizer to mitigate channel distortion during OTA aggregation. A reference combiner forms an effective scalar channel, while feature projection provides symbols across subcarriers.

1) Model-Driven OTA Alignment Symbol Mapping:

  • The MMSE-inspired model-driven mapping uses a learnable linear combiner and robust pre-equalizer to mitigate channel distortion in OTA aggregation.Its regularized channel-conjugate structure accounts for imperfect CSI and reduces to the standard form when estimation error is zero.Source: 1) Model-Driven OTA Alignment Symbol… · pp. 7–8
  • A unit-norm reference combining vector forms an effective scalar channel from each user's physical channel for transmitter optimization.The combining vector is jointly optimized with the other network parameters during training.Source: 1) Model-Driven OTA Alignment Symbol… · p. 7
  • The channel-aware feature is projected through learnable subcarrier vectors to form per-user symbols for transmission.The projection is trained end-to-end to adapt to frequency-selective features.Source: 1) Model-Driven OTA Alignment Symbol… · p. 7
  • The model-driven branch phase-conjugates symbols to align them with the effective channel for constructive combining.Its regularization balances multi-user channel power against noise power and channel uncertainty.Source: 1) Model-Driven OTA Alignment Symbol… · p. 8

2) Data-Driven Feature-Aware Beamforming:

The data-driven branch uses a Transformer to map features to transmission symbols. Feature attention, embedding, and positional encoding prepare inputs for modeling dependencies across subcarriers.

2) Data-Driven Feature-Aware Beamforming:

3) Hybrid Symbol Mapping:

Hybrid symbol mapping combines data-driven adaptability with model-driven robustness through a learned weighted sum. Softmax-constrained weights are trained jointly with the network to set the balance.

3) Hybrid Symbol Mapping:

  • A learnable weighted sum fuses the data-driven and model-driven symbol mapping strategies.The fusion is designed to retain data-driven adaptability and model-driven robustness.Source: 3) Hybrid Symbol Mapping · p. 8
  • Softmax parameterization constrains the two mapping weights to sum to one and remain between zero and one.The weights are determined by learnable scalars θd and θm.Source: 3) Hybrid Symbol Mapping · p. 8
  • End-to-end training jointly optimizes the mapping weights with the other network parameters, adapting their balance to the task objective and channel conditions.Source: 3) Hybrid Symbol Mapping · p. 8

4) Power Normalization:

The transmitter normalizes each fused symbol vector to meet its user's power budget. On the receiver side, OTA signals form a SIM input that is physically transformed into class outputs and trained with task and GNN losses.

4) Power Normalization:

  • After hybrid fusion, each user's symbol vector is normalized to satisfy its power budget while preserving learned relative amplitude and phase relationships.The budget is denoted Pk for user k.Source: 4) Power Normalization · p. 8

D. OTA Aggregation and CI

  • Simultaneously transmitted user symbols propagate through their multipath channels and superimpose at the server as the received space-frequency matrix Y.The matrix has Nc rows and Nr antenna dimensions per subcarrier.Source: D. OTA Aggregation and CI · p. 9
  • The receiver preprocesses Y to fit the SIM input size and vectorizes its aperture samples into yvec, with M = Nc · Nr.Source: D. OTA Aggregation and CI · p. 9
  • The SIM is modeled as a trainable C × M transformation physically realized by its differentiable stack of propagation and meta-atom layers.The meta-atoms' amplitude and phase parameters are trainable through back-propagated gradients.Source: D. OTA Aggregation and CI · p. 9
  • The end-to-end trained SIM jointly performs spatial-frequency combining and feature classification in a single physical wave propagation.The SIM processes yvec into output signals whose port energies serve as class logits.Source: D. OTA Aggregation and CI · p. 9
  • For multi-label inference, sigmoid converts class logits to independent probabilities, and the task loss is binary cross-entropy combined with GNN self-supervised losses.Source: D. OTA Aggregation and CI · p. 9

V. Performance Results and Analysis

The evaluation uses COCO2017 and SpokenCOCO for multimodal collaborative inference under a simulated multi-user OFDM uplink. The section specifies the datasets, channel and network settings, training configuration, and mAP metric.

A. Experimental Setting

  • COCO2017 provides 118,287 training images and 5,000 validation images across 80 categories, while SpokenCOCO supplies aligned speech, image, and text samples for tri-modal evaluation.Source: A. Experimental Setting · pp. 9–10
  • The simulated uplink uses four single-antenna users, 64 OFDM subcarriers, and a 64-antenna receiver array over multipath Rayleigh fading channels.The carrier frequency is 2.6 GHz, subcarrier spacing is 60 kHz, and each user has two propagation paths.Source: A. Experimental Setting · p. 10
  • Visual and textual features are extracted with Faster R-CNN/ResNet-101 and BERT-base-uncased, then projected to a shared 1024-dimensional space.Source: A. Experimental Setting · p. 10
  • The cross-modal GNN uses one layer, and the data-driven beamforming branch uses a four-layer Transformer; training uses Adam at 1 × 10^-4 with batch size 128.Intra-modal and cross-modal adapter losses each receive weight 0.5; contrastive and symmetry losses are also weighted 0.5 each.Source: A. Experimental Setting · p. 10
  • Mean Average Precision (mAP) is the primary metric for multi-label classification accuracy.Source: A. Experimental Setting · p. 10

B. SIM-CI Accuracy and Computational Overhead

SIM-CI nearly matches BLIP-2's reported mAP while reducing several overhead measures in the stated simulation. The channel ablations report gains from channel-aware features and hybrid beamforming, alongside robustness evaluations across SNR, transmit power, and model depth.

B. SIM-CI Accuracy and Computational Overhead

  • The SIM model has O(L · N) trainable diagonal response parameters, compared with O(L · N^2) dense trainable weights and biases in the equivalent-depth DNN.Source: B. SIM-CI Accuracy and Computational… · p. 10
  • For multimodal receiver energy, SIM-CI uses 0.0060K versus BLIP-2's 0.0080K because both share the adapter and differ at the receiver.For image-only inference, SIM-TSC uses 0.0042K versus ViT's 0.0045K despite SIM-TSC's adapter; the passage attributes this to the DNN receiver cost exceeding the SIM adapter cost.Source: B. SIM-CI Accuracy and Computational… · p. 10
  • SIM-CI reaches 0.723 mAP versus BLIP-2's 0.733, nearly matching the multimodal digital baseline; both outperform the four single-modality baselines.The compared single-modality methods are SIM-TSC, Adapter-BERT, ViT, and RoBERTa.Source: B. SIM-CI Accuracy and Computational… · p. 10, Figure 3 · p. 11
  • Digital methods use K separate TDMA slots, whereas SIM-based OTA methods transmit users simultaneously; as M grows, digital receiver latency rises linearly while SIM latency stays constant.Source: B. SIM-CI Accuracy and Computational… · p. 11

C. Robustness to Channel Heterogeneity

  • Across 6–15 dB SNR, the channel-aware GNN configuration outperforms fading-only, and full SIM-CI performs best; at 10 dB, their reported mAP values include 0.672, 0.662, and 0.706, respectively.The ablation compares fading-only, GNN-aware without beamforming, and the complete SIM-CI framework.Source: Figure 4 · p. 11, C. Robustness to Channel Heterogeneity · p. 11
  • At 30 dBm, fading-only plateaus near 0.698 mAP, compared with approximately 0.708 for GNN-aware and 0.717 for SIM-CI.The passage attributes the full framework's gain to hybrid beamforming's amplitude pre-equalization and phase pre-alignment.Source: Figure 4 · p. 11, C. Robustness to Channel Heterogeneity · pp. 11–12

B. SIM-CI Accuracy and Computational Overhead

C. Robustness to Channel Heterogeneity

  • Fig. 5 evaluates robustness to model-depth heterogeneity as the proportion of mid-level feature users increases.Source: Figure 5 · p. 12

D. Robustness to Model-Depth Heterogeneity

Across model-depth and modality heterogeneity tests, SIM-CI remains robust as shallow-feature users increase and achieves the highest mAP in every evaluated modality configuration. The conclusion reports 98.6% of conventional digital-domain CI accuracy, with 25% less computational energy and 75% fewer transmission time slots.

D. Robustness to Model-Depth Heterogeneity

  • SIM-CI remains the most robust as shallow-feature users increase, reaching 0.598 mAP at ps = 1 while GAT falls to 0.537.The test fixes K = 4 and averages over possible depth assignments; the adapter is trained jointly on shallow, mid, and deep features, with quality-aware contrastive learning aligning extraction depths.Source: Figure 6 · p. 12, D. Robustness to Model-Depth… · pp. 12–13

E. Performance under Modality Heterogeneity

  • SIM-CI achieves the highest mAP in every modality configuration; Image+Text reaches 0.714 mAP and Image+Text+Speech reaches 0.718 mAP.Its advantage over baselines is most pronounced with multiple modalities and grows in multimodal and mixed configurations; the adapter models cross-modal relations to map heterogeneous client types into a common semantic space.Source: Table II · p. 13, E. Performance under Modality… · p. 13

VI. Conclusion

  • 98.6% of the accuracy of conventional digital-domain CI approaches, while reducing computational energy by 25% and requiring 75% fewer transmission time slots.The conclusion presents SIM-CI as a physical-domain collaborative inference framework for heterogeneous conditions.Source: §VI · p. 13
Loading 2610.13364v1…