PaperlayerView exact revision on arXiv

Source-linked AI summary

Large AI Model-Based Semantic Communications

Feibo Jiang, Yubo Peng, Li Dong, Kezhi Wang, Kun Yang, Cunhua Pan, Xiaohu You

arXiv:2307.03492v2cs.AIcs.NI

TL;DR

Current SC knowledge bases face limits in representation, costly updates, and insecure sharing. LAM-SC combines SAM-based segmentation with attention-based integration and adaptive semantic compression for image transmission. Simulations report best performance on VOC2012 and COCO2017, with similar SSIM using 55% of SC's transmitted data.

  • Problem

    Current SC knowledge bases have limited knowledge representation, require costly updates, and may expose sensitive information through knowledge sharing.

    Source: B. Issues about Current KB Schemes in… · p. 2

  • Method

    LAM-SC uses SAM as an image knowledge base to segment images, ASI to integrate important segments, and ASC to remove redundant semantic features.

    Source: C. Our Contributions · p. 2, 2) SAM-Based KBs · p. 3

  • Results

    LAM-SC achieves the best SSIM performance on VOC2012 and COCO2017; at similar SSIM, it transmits 55% of SC's data.

    Source: §V · p. 7

  • Takeaways & Limitations

    The simulations demonstrate the effectiveness of LAM-SC for image transmission and the possibility of applying LAM-based knowledge bases in future SC paradigms.

    Source: §VII · p. 7

    LAM implementation requires significant energy compared with traditional methods, raising environmental concerns and accessibility challenges for mobile and edge devices.

    Source: B. Energy Consumption for Mobile Devices · p. 7

Abstract

from arXiv · show

Semantic communication (SC) is an emerging intelligent paradigm, offering solutions for various future applications like metaverse, mixed reality, and the Internet of Everything. However, in current SC systems, the construction of the knowledge base (KB) faces several issues, including limited knowledge representation, frequent knowledge updates, and insecure knowledge sharing. Fortunately, the development of the large AI model (LAM) provides new solutions to overcome the above issues. Here, we propose a LAM-based SC framework (LAM-SC) specifically designed for image data, where we first apply the segment anything model (SAM)-based KB (SKB) that can split the original image into different semantic segments by universal semantic knowledge. Then, we present an attention-based semantic integration (ASI) to weigh the semantic segments generated by SKB without human participation and integrate them as the semantic aware image. Additionally, we propose an adaptive semantic compression (ASC) encoding to remove redundant information in semantic features, thereby reducing communication overhead. Finally, through simulations, we demonstrate the effectiveness of the LAM-SC framework and the possibility of applying the LAM-based KB in future SC paradigms.

I. INTRODUCTION

Semantic communication prioritizes delivering intended meaning with minimal data, using a knowledge base to help the encoder and decoder understand and infer semantics. A universal KB includes both structured prior knowledge and contextual background knowledge.

I. INTRODUCTION

  • SC prioritizes delivering intended meaning with minimal data, unlike traditional systems that focus on accurate transmission of bits or symbols.Its semantic encoder extracts meaning-bearing features to reduce transmitted information.Source: §I · p. 1
  • A universal knowledge base helps semantic encoders and decoders understand and infer semantic information.SC components can use jointly trained DNNs to improve semantic faithfulness and reduce communication overhead.Source: §I · p. 1
  • AI-powered SC models including TOSCN, DeepSC-ST, and DeepJSCC-V emphasize communication-model design but may not fully exploit KBs.Their approaches include encoder-decoder, information bottleneck, and knowledge graph architectures.Source: §I · p. 1

A. Composition of a Universal KB in SC Systems

  • Prior knowledge represents entities, attributes, and relationships, while background knowledge contributes context, implicit meaning, and common facts.Together, these knowledge types support shared interpretation and help extract relevant information while eliminating redundancy.Source: A. Composition of a Universal KB in SC… · pp. 1–2

B. Issues about Current KB Schemes in SC Systems

Current SC knowledge bases face limits in knowledge representation, costly updates, and insecure sharing. The paper recommends LAM-based KBs and presents an image-transmission framework combining SKB, ASI, and ASC.

B. Issues about Current KB Schemes in SC Systems

  • Traditional SC knowledge bases have limited representation, costly updates as domains change, and privacy and security risks from sharing sensitive knowledge models.Limited parameters and labeled data constrain learned semantic knowledge; frequent training and sharing consume resources.Source: B. Issues about Current KB Schemes in… · p. 2

C. Our Contributions

  • LAMs are recommended for building KBs, and the paper explores LAM-based schemes for different SC systems.For image transmission, SKB segments images, ASI weighs and integrates segments without human intervention, and ASC adaptively masks redundant semantic features.Source: C. Our Contributions · p. 2

II. LAMS-BASED KBS IN SC SYSTEMS

The paper describes LAM-based KBs as offering rich knowledge and low-cost updates, then sketches uses across SC modalities. For text, ChatGPT can extract requested content and help reduce semantic noise at the receiver.

A. Advantages of LAMs-Based KBs

  • LAMs are described as offering accurate knowledge representation, rich prior and background knowledge, and low-cost knowledge updates.The paper attributes these advantages to extensive pretraining and the ability to prompt or fine-tune with limited labeled data.Source: A. Advantages of LAMs-Based KBs · p. 3

B. Design Suggestions of LAMs in SC Systems

2) SAM-Based KBs:

The paper presents an image-focused LAM-SC framework using SAM as a knowledge base, alongside possible LAM-based KB designs for audio and text communication. It contrasts these application areas and notes the limited research on image-based SC.

2) SAM-Based KBs:

  • SAM generalizes zero-shot segmentation to unfamiliar images and objectives without additional training; the sender can select segments for encoding, and SAM can help identify pertinent segments after image restoration.The proposed image framework incorporates SAM as its KB.Source: 2) SAM-Based KBs · p. 3

3) WavLM-Based KBs:

  • WavLM can separate and recognize different speakers, discard background noise, and support receiver-side speech denoising and recognition according to user requirements.The passage describes WavLM as trained on 94,000 hours of unsupervised English data.Source: 3) WavLM-Based KBs · p. 3
  • WavLM-based SC targets real-time audio interaction, GPT-based SC conveys ideas through text, and SAM-based SC represents visual information; image-based SC has relatively little research.Source: 3) WavLM-Based KBs · p. 3

III. ARCHITECTURE OF LAM-SC FRAMEWORK

A. Introduction to LAM-SC Framework

The framework segments images into individual semantic objectives, selects and integrates segments into a semantic-aware image, and encodes the result for transmission. At the receiver, it decodes the image and applies the knowledge base again to identify objectives.

1) KB Construction and Semantic Segmentation:

2) Attention-Based Semantic Integration:

  • ASI uses channel and spatial attention to select segments of interest, with human prompts offering a way to specify selections; the selected segments are merged into a semantic-aware image.Source: 2) Attention-Based Semantic Integration · p. 4

3) Semantic Adaptive Encoding and Channel Encoding:

  • A CNN-based semantic encoder extracts image features, ASC masks unimportant semantic features, and an MLP-based channel encoder encodes and modulates signals for the physical channel.Source: 3) Semantic Adaptive Encoding and… · p. 4

4) Channel Decoding and Semantic Decoding:

  • At the receiver, an MLP channel decoder recovers semantic features, a deconvolution-based semantic decoder reconstructs the image, and SKB identifies objectives in the recovered image.Source: 4) Channel Decoding and Semantic… · p. 4

B. SKB

The SKB uses SAM for zero-shot image segmentation, while ASI applies channel and spatial attention to select important segments. The paper states that ASI can retain objects of greater interest to humans without human involvement.

B. SKB

  • SAM-based SKB performs image segmentation without additional training, generalizing zero-shot to unseen images or objectives; SAM's training dataset contains over 1 billion masks across 11 million images [12].Source: B. SKB · p. 5

1) Channel Attention Network:

  • Channel attention treats each semantic segment as a channel, uses max and average pooling, and combines MLP outputs to weight segment importance.Source: 1) Channel Attention Network · p. 5

2) Spatial Attention Network:

  • Spatial attention pools channel-weighted semantics, concatenates the results, and uses a linear-layer output head to select segments for integration into semantic-aware images.Source: 2) Spatial Attention Network · p. 5

C. ASI

D. ASC

ASC masks unimportant semantic features to reduce redundant transmitted data and communication overhead. The framework's training description also explains how human-interested semantics inform the attention networks.

D. ASC

  • A learnable mask network outputs a binary mask that sets some unimportant encoded semantic features to zero before transmission.Source: D. ASC · p. 5
  • ASC maintains essential semantic features while excluding redundant ones, thereby reducing communication overhead.Source: D. ASC · p. 5

1) ASI Training Based on Human Experience:

  • ASI training uses semantic segments as inputs and human-prompt selections as labels, applying supervised learning so attention networks can make decisions resembling human perception.The framework uses SAM in SKB as a pre-trained LAM.Source: 1) ASI Training Based on Human… · p. 5, §IV · p. 5

2) Crossover-Based SC Encoder and Decoder Training:

The section describes alternating channel/semantic-model training and mask-network training for ASC, then reports that LAM-SC leads the compared schemes while transmitting less data at similar SSIM.

2) Crossover-Based SC Encoder and Decoder Training:

  • Training alternates between the channel encoder/decoder and semantic encoder/decoder, freezing one model pair while training the other until convergence.The channel pair is trained first, followed by the semantic pair, then the channel pair again.Source: 2) Crossover-Based SC Encoder and… · p. 5

3) ASC Training:

  • ASC trains the mask network to minimize reconstruction difference while the SC model and attention networks are frozen.Masked semantic features are transmitted, the image is recovered, and reconstruction difference guides mask-network training.Source: 3) ASC Training · pp. 5–6

V. SIMULATION RESULTS

  • At SNR = 5 dB, LAM-SC has the lowest system loss and achieves the best SSIM on VOC2012 and COCO2017.Source: §V · pp. 6–7, Figure 4 · p. 6, Figure 5 · p. 7
  • LAM-SC transmits 8,960 bits versus 21,632 bits for traditional SC semantic features, and its data volume is 55% of SC's at similar SSIM.The original image requires 49,152 bits to transmit.Source: §V · p. 7

VI. OPEN ISSUES AND FUTURE DIRECTIONS

The discussion identifies latency and bandwidth as constraints on using large AI models in communication systems, particularly for real-time applications.

A. Latency in Real-Time Applications

B. Energy Consumption for Mobile Devices

The section identifies energy consumption, interpretability, and privacy as challenges for LAM-based SC, then concludes with an image-focused LAM-SC framework using SKB, ASI, and ASC.

B. Energy Consumption for Mobile Devices

  • LAM implementation in SC consumes significantly more energy than traditional methods, raising environmental concerns and accessibility challenges for mobile and edge devices.Sparse attention, quantization, and distillation are suggested as possible ways to reduce model size and energy consumption.Source: B. Energy Consumption for Mobile Devices · p. 7

C. Explainability and Transparency

  • Interpreting LAM-produced KB and semantic-encoder outputs can be difficult, complicating the identification of potential biases or errors in SC systems.Fuzzy rules and visual heat maps are suggested as possible explainable-AI directions.Source: C. Explainability and Transparency · p. 7

D. Privacy and Security

  • LAMs may capture sensitive information during training or infer sensitive details from processed data, raising privacy, security, and ethical concerns in SC.Suggested directions include federated learning for fine-tuning and privacy technologies during parameter or information exchange.Source: D. Privacy and Security · p. 7

VII. CONCLUSION

  • The conclusion proposes image-focused LAM-SC: SAM-based SKB segments images, ASI integrates segment semantics, and ASC reduces communication overhead; simulations demonstrate the framework’s effectiveness.The paper also discusses the possibility of applying LAM-based KBs in future SC paradigms.Source: §VII · p. 7

BIOGRAPHIES

The biographies summarize the authors’ academic backgrounds, current appointments, and research interests.

BIOGRAPHIES

  • Feibo Jiang, Yubo Peng, and Li Dong list academic training and appointments alongside interests spanning AI, machine learning, IoT, mobile edge computing, and federated learning.Their profiles also identify language processing, fuzzy computation, and semantic communication among their interests.Source: BIOGRAPHIES · p. 8
  • Kezhi Wang, Kun Yang, and Cunhua Pan describe academic appointments and research interests in wireless communications, machine learning, IoT, and mobile-edge computing.Their listed interests also include RIS, low-latency communications, UAVs, and data- and energy-integrated networks.Source: BIOGRAPHIES · p. 8
  • Xiaohu You’s biography records his long-term roles in China’s mobile communications research and honors, alongside interests in mobile systems and signal processing.The profile notes his 2011 IEEE Fellow selection and National 1st Class Invention Prize.Source: BIOGRAPHIES · pp. 8–9
Loading 2307.03492v2…