Source-linked AI summary
Multi-Source and Multi-Dimensional Information Fusion in Resource-Constrained Edge Networks
Yushi Wang, Zhengxin Yu, Guhan Zheng, Wang Miao, Merouane Debbah
TL;DR
Limited communication and computation resources leave edge servers with incomplete, heterogeneous semantic tokens whose value depends on other terminals' contributions. The paper jointly coordinates uploading and fusion, using Hyper-Slot and a contribution-based game. Hyper-Slot outperforms evaluated baselines on both reported tasks, with higher mAP and RSum than the strongest baselines.
Problem
Resource constraints leave edge servers with incomplete heterogeneous tokens, and effective fusion must account for token attributes and the information supplied by other terminals.
Source: §I · pp. 1–2Method
The framework coordinates selective uploading with Hyper-Slot, which assigns tokens to latent slots using semantic content and attributes, and uses fusion-based contributions in its uploading incentive mechanism.
Source: §I · p. 2, B. Hyper-Slot for Multi-dimensional… · p. 5Results
Hyper-Slot achieves the highest mean mAP (0.7610±0.0239), above the strongest baselines (SCAN and Slot Attention, both at 0.724), and the highest mean RSum (478.9±3.4), exceeding Slot Attention (463.0 ± 3.2) by 15.9 points.
Source: B. Performance Comparison with Other… · p. 9, F. Performance Evaluation on Flickr30K · p. 11Takeaways & Limitations
The framework links terminal uploading decisions to edge-side fusion, with rewards based on each terminal’s marginal contribution to fusion gain.
Source: §I · p. 2, 2) Utility Function of the Edge Server · p. 8
Abstract
from arXiv · showhide
Multi-source semantic information fusion enables edge networks to support global inference using observations collected by distributed terminals. However, limited communication and computation resources restrict semantic uploading, leaving the edge server with incomplete tokens that differ in modality, semantic granularity, arrival time, and channel condition. The value of these tokens also depends on the complementary or redundant information provided by other terminals. In this paper, we propose a multi-source and multi-dimensional semantic information fusion framework that coordinates terminal uploading and edge-side fusion under resource constraints. We develop Hyper-Slot to combine semantic content with available token attributes through iterative token-to-slot assignment and slot-to-token feedback, producing a compact representation for global inference. To encourage useful semantic uploading, we further design a contribution-based incentive mechanism that uses Hyper-Slot assignments to evaluate information coverage and cross-type complementarity. A bi-level Stackelberg-Nash game determines the server reward pool and terminal transmission-quality strategies while accounting for computation and communication energy costs. Experiments on a tri-modal extension of COCO2014 and image-text retrieval on Flickr30K show that Hyper-Slot improves mAP by 3.66 percentage points and RSum by 15.9 points over the strongest evaluated baselines, respectively, while reducing the number of parameters by more than 80% compared with a naive concatenation-based fusion baseline.
I. Introduction
Resource-constrained edge networks receive incomplete semantic tokens whose value depends on their attributes and on what other terminals contribute. The paper motivates a framework that jointly coordinates selective uploading and edge-side fusion.
I. Introduction
- Limited computation, transmission power, and bandwidth prevent terminals from always uploading complete, high-quality semantic information.Independent upload decisions may omit important evidence or send redundant information, reducing the coverage and complementarity available at the edge server.Source: §I · p. 1
- The framework jointly coordinates selective semantic uploading and edge fusion under communication and computation constraints.It connects fusion assignments to terminal contribution evaluation and uploading decisions.Source: §I · p. 2
- Hyper-Slot uses semantic content and available token attributes in iterative token-to-slot assignment and slot-to-token feedback to form a compact task-level representation.The incentive mechanism evaluates contributions through Hyper-Slot assignments, accounting for information coverage, cross-type complementarity, and computation and communication energy costs.Source: §I · p. 2
II. Related Works
- Prior resource-constrained semantic communication work optimizes delivery, but leaves the value of transmitted tokens for subsequent multi-source fusion unmodeled.Existing fusion approaches also tend to assume complete uploads, while slot-based methods such as Slot Attention do not address communication, heterogeneous attributes, or incomplete inputs.Source: §II · pp. 2–3
A. Multi-Source Semantic Communication and Fusion Model
- Terminals generate candidate tokens from local observations, but resource limits restrict uploads and leave the server with an incomplete, variable-size set that may be complementary or overlapping.Tokens share a semantic space but can differ in content, attribute configuration, and size; descriptors may record source, modality, granularity, temporal state, and channel condition.Source: A. Multi-Source Semantic Communication… · pp. 3–4
B. Computation and Communication Overhead Model
The overhead model accounts for terminal-side computation and communication costs associated with uploading selected semantic tokens.
B. Computation and Communication Overhead Model
- Uploading selected semantic tokens incurs both computation and communication overhead at each terminal.The model tracks processing delay and energy alongside communication delay and energy.Source: B. Computation and Communication… · pp. 4–5
- Computation load is based on the CPU cycles required to process each selected token for uploading.Processing may include token refinement, attribute processing, and transmission preparation; the model also gives computation delay and energy.Source: B. Computation and Communication… · p. 4
- Communication overhead is modeled from the total size of uploaded tokens and the achievable uplink rate.The model derives communication delay and energy, then combines the computation and communication costs into total processing delay and energy.Source: B. Computation and Communication… · pp. 4–5
IV. The Proposed Framework
The proposed framework addresses multi-source, multi-dimensional semantic fusion in resource- and communication-constrained edge networks. This section introduces the framework, Hyper-Slot, and a semantic-uploading mechanism based on a bi-level Stackelberg–Nash game.
IV. The Proposed Framework
- The proposed framework targets multi-source and multi-dimensional semantic information fusion in resource- and communication-constrained edge networks.It is organized around an overall framework, the Hyper-Slot fusion module, and a game-based semantic-uploading mechanism.Source: §IV · p. 5
- Hyper-Slot is the framework's designated fusion module for multi-dimensional semantic information.The section introduces this module after describing the overall framework.Source: §IV · p. 5
- The semantic-uploading mechanism is based on a bi-level Stackelberg–Nash game.The section presents it alongside the Hyper-Slot fusion module.Source: §IV · p. 5
A. Framework Description
The framework selects tokens at terminals, coordinates uploading through a two-level game, and fuses received tokens at the edge server with Hyper-Slot. The fusion assignments also inform terminal rewards based on marginal contribution.
A. Framework Description
- Each terminal generates candidate tokens from its local observation and selects uploads under its own communication and computation constraints.Source: A. Framework Description · p. 5
- Received tokens may be incomplete and variable in size, with heterogeneity in modality, semantic granularity, arrival time, and channel condition.Hyper-Slot combines token content with available attributes to produce a unified task-level representation.Source: A. Framework Description · p. 5
- The server announces an incentive, after which terminals choose uploading strategies through a non-cooperative game that settles at a Nash equilibrium.Each terminal's reward is based on its marginal contribution to fusion gain, computed from Hyper-Slot assignment weights.Source: A. Framework Description · p. 5
- The two-level game determines which information reaches the fusion stage, and Hyper-Slot determines how the selected information is combined into the global inference result.The process begins with candidate token generation and ends with edge-side fusion.Source: A. Framework Description · p. 5
B. Hyper-Slot for Multi-dimensional Fusion
Hyper-Slot fuses incomplete, heterogeneous semantic tokens by assigning them to a fixed set of latent slots using both semantic content and task-relevant attributes. Iterative refinement and feedback produce a pooled representation for downstream prediction, with diversity regularization to discourage slot collapse.
B. Hyper-Slot for Multi-dimensional Fusion
- Hyper-Slot maps heterogeneous tokens into a fixed number of latent slots, each adaptively grouping tokens coherent in semantic and task-specific dimensions.The heterogeneity dimensions can be configured for the edge task rather than fixed by the model.Source: B. Hyper-Slot for Multi-dimensional… · pp. 5–6
- The heterogeneity descriptor can encode categorical or continuous task-configurable properties, including source, modality, granularity, confidence, freshness, or communication condition.Categorical and continuous dimensions are mapped into a unified descriptor space.Source: B. Hyper-Slot for Multi-dimensional… · p. 6
- Token-to-slot affinity combines semantic similarity with heterogeneity consistency, while a descriptor-dependent gate modulates each token’s contribution.The gate reflects token reliability or usefulness under its heterogeneity descriptor.Source: B. Hyper-Slot for Multi-dimensional… · p. 6
- Recurrent slot refinement is followed by writing slot information back to token states, allowing tokens to adjust assignments across iterations.The final slots are pooled into a fused representation and passed to a task-specific prediction head.Source: B. Hyper-Slot for Multi-dimensional… · p. 6
- Training combines the downstream task loss with diversity regularization over heterogeneity anchors to prevent slots from collapsing to similar patterns.The regularization strength is controlled by λ_div.Source: B. Hyper-Slot for Multi-dimensional… · p. 6
C. Incentive Mechanism for Semantic Uploading
The incentive mechanism values terminal uploads through their marginal fusion contributions, measured using information coverage and cross-type complementarity. A server-set reward pool and terminals’ transmission-quality strategies are determined through a coupled Stackelberg–Nash game that accounts for energy costs.
C. Incentive Mechanism for Semantic Uploading
- The server announces a reward pool before uploading, then anticipates the terminals’ Nash-equilibrium strategies when choosing that pool.The terminal strategies are coupled because each affects fusion gain and contribution-based reward allocation.Source: C. Incentive Mechanism for Semantic… · p. 6, 2) Utility Function of the Edge Server · p. 8, 3) Bi-level Stackelberg–Nash Game… · p. 8
- Each terminal selects token-level transmission quality to balance contribution-based rewards against computation and communication energy costs.A quality of zero means no transmission; positive values represent normalized link quality, subject to a terminal-wide resource budget.Source: 1) Utility Function of Terminals · p. 7
- A terminal’s contribution is the marginal change in fusion gain when it participates, conditioned on the information jointly provided by the other terminals.This contribution is used to allocate the announced reward pool among terminals.Source: C. Incentive Mechanism for Semantic… · p. 6, 1) Utility Function of Terminals · p. 7
- Fusion gain combines effective information coverage within slots with complementarity between different information types assigned to the same slot.Coverage uses a saturating function, while the complementarity between two types is limited by the type with lower coverage.Source: 1) Utility Function of Terminals · p. 7
- The coupled optimization is treated as a bi-level Stackelberg–Nash game, with the server as leader and terminals playing a non-cooperative game.The proposed algorithm uses damped best-response iteration for terminals and numerical gradient ascent to search for the server’s reward pool.Source: 3) Bi-level Stackelberg–Nash Game… · p. 8
V. Simulations
The simulations evaluate semantic fusion, cross-modal retrieval, and the semantic uploading mechanism, including comparisons of Hyper-Slot with other fusion methods.
V. Simulations
- The simulations compare fusion methods and examine heterogeneity dimensions, transmission budgets, structural components, cross-modal retrieval, and semantic uploading.The evaluation includes semantic fusion experiments and game-theoretic simulations.Source: §V · p. 9
A. Simulation Settings
The simulation settings use a tri-modal extension of COCO2014 with vision, text, and audio sources, and instantiate semantic granularity, arrival time, and channel condition as additional heterogeneity dimensions.
A. Simulation Settings
- The tri-modal COCO2014 extension contains 30,000 training pairs and 5,000 validation pairs across 80 categories, pairing images with original captions and corresponding spoken captions.The three sources are vision, text, and audio.Source: A. Simulation Settings · p. 9
- Visual region features, BERT text-token features, and HuBERT audio-segment features are projected into a unified embedding dimension.Audio segments are aligned to word-level time boundaries before feature extraction.Source: A. Simulation Settings · p. 9
- The token descriptors instantiate modality, semantic granularity, arrival time, and channel quality, with the latter two simulated to represent asynchronous reporting and time-varying wireless conditions.Hyper-Slot uses 16 slots and three refinement iterations, with 256-dimensional token, slot, and descriptor representations.Source: A. Simulation Settings · p. 9
B. Performance Comparison with Other Fusion Methods
Hyper-Slot outperforms representative fusion baselines on overall and per-class mAP, while learning token groups using semantic similarity and heterogeneity consistency.
B. Performance Comparison with Other Fusion Methods
- Hyper-Slot combines semantic similarity with heterogeneity consistency for slot assignment, unlike baselines that use pooling, pairwise attention, or predefined grouping rules.Its assignments are refined through iterative slot-to-token feedback.Source: B. Performance Comparison with Other… · p. 9
- 0.7610±0.0239 mean mAP over five seeds outperformed the strongest baselines, SCAN and Slot Attention, both at 0.724.Hyper-Slot used more parameters and GFLOPs than Slot Attention or Static HGNN, but was substantially lighter than Concat + MLP.Source: B. Performance Comparison with Other… · p. 9
- Hyper-Slot achieved the highest mAP in each of five representative categories, and tokens of the same modality occupied multiple slots.Each slot anchor lies close to its assigned token cluster.Source: B. Performance Comparison with Other… · p. 9
C. Effect of Heterogeneity Dimensions
The full heterogeneity descriptor outperforms configurations using content alone or individual dimensions. The results indicate that the dimensions complement one another and affect seed-to-seed stability.
C. Effect of Heterogeneity Dimensions
- The six configurations were content alone, one descriptor dimension at a time, or the full descriptor of modality, semantic granularity, arrival time, and channel quality.mAP was evaluated under each configuration across five random seeds.Source: C. Effect of Heterogeneity Dimensions · pp. 9–10
- The full four-dimensional descriptor achieved the highest mean mAP across seeds, exceeding every single-dimension configuration by a substantial margin.Content-only had the lowest mean mAP among the six configurations; the dimensions were complementary rather than redundant.Source: Figure 5 · p. 10, C. Effect of Heterogeneity Dimensions · p. 10
- Single-dimension configurations showed markedly larger seed-to-seed variance than full heterogeneity.No single attribute recovered the performance achieved by combining all four dimensions.Source: C. Effect of Heterogeneity Dimensions · p. 10
D. Effect of Transmission Budget
Task-aware token selection performs best below the full-transmission budget, using Hyper-Slot reliability and coverage signals to rank tokens. Its advantage narrows as the budget increases.
D. Effect of Transmission Budget
- Task-aware selection ranks tokens using Hyper-Slot’s descriptor-dependent gate as a reliability proxy and assignment weight as a proxy for marginal slot coverage.The alternatives are Random, Top-SNR, and Top-importance selection.Source: D. Effect of Transmission Budget · pp. 10–11
- At every budget below 100%, task-aware selection dominated; its largest margin over Top-importance was at 25%, and it converged to the full-transmission upper bound at 100%.Figure 6 compares mAP at 25%, 50%, 75%, and 100% of full-transmission cost.Source: Figure 6 · p. 11, D. Effect of Transmission Budget · p. 11
- Below 100% budget, Top-importance consistently outperformed Random and Top-SNR selection.Top-SNR prioritizes channel conditions regardless of semantic value, while Random ignores content.Source: D. Effect of Transmission Budget · pp. 10–11
E. Structural Ablation of Hyper-Slot
The section evaluates Hyper-Slot’s structural components, Flickr30K retrieval, and incentive-game behavior, then summarizes the framework’s benchmark gains. The results show complementary fusion components and contribution-differentiated rewards.
E. Structural Ablation of Hyper-Slot
- The full model reaches ≈0.75 mAP, versus ≈0.72 for compact only and ≈0.665 for dependency only; removing the compact bottleneck causes the larger performance drop.Both variants converge within 20–30 epochs, but neither matches the full model; the two design elements are complementary.Source: E. Structural Ablation of Hyper-Slot · p. 11, Figure 7 · p. 11
F. Performance Evaluation on Flickr30K
- On Flickr30K, Hyper-Slot’s mean RSum is 478.9±3.4, 15.9 points above Slot Attention’s 463.0 ± 3.2 over five random seeds.It has fewer parameters than the concatenation baseline and Slot Attention while achieving the highest accuracy among the seven methods.Source: F. Performance Evaluation on Flickr30K · p. 11, Table II · p. 12
G. Performance of the Incentive Uploading Mechanism
- At R*=4.91, server utility peaks as larger rewards increase resource utilization and fusion benefit, while system cost rises and eventually outweighs additional benefit.The fusion-benefit gain becomes progressively smaller due to saturation.Source: G. Performance of the Incentive… · p. 12, Figure 8 · p. 12
- Terminal 4 has the largest semantic contribution and incentive, while Terminal 1 has the smallest; total allocated incentives satisfy P_m≈R*.The incentive distribution follows semantic contributions, which reflect additional information for global fusion rather than simply upload amount.Source: G. Performance of the Incentive… · p. 12, Figure 9 · p. 13
VI. Conclusion
- Hyper-Slot exceeds the strongest evaluated baselines by 3.66 percentage points in mAP on tri-modal COCO2014 and 15.9 points in RSum on Flickr30K, with fewer than one-fifth as many parameters as naive concatenation.Under a constrained 25% transmission budget, it also sustains a 5.46-percentage-point mAP advantage over token selection.Source: §VI · p. 13