Source-linked AI summary
Foundation Models for Generalizable Semantic and Goal-Oriented Communication
Boliang Liu, Wint Yi Poe, Riccardo Trivisonno, Giuseppe Caire
TL;DR
Existing end-to-end semantic communication systems generalize poorly beyond training data and degrade sharply at very low rates because they compress the entire signal. FMSGOC decouples semantic selection from reconstruction by transmitting sparse anchors chosen by a VLM and completing masked regions with a LoRA-tuned diffusion model. It achieves robust semantic fidelity and zero-shot generalization at 0.039 BPP, outperforming end-to-end baselines at that rate.
Problem
Coupled end-to-end systems generalize poorly to unseen content and degrade sharply at ultra-low bitrates when critical semantics cannot be retained.
Method
FMSGOC uses a VLM to select sparse semantic anchors and a LoRA fine-tuned diffusion model to reconstruct the image from those anchors.
Results
0.039 BPP: FMSGOC demonstrates robust semantic fidelity and zero-shot generalization, while end-to-end baselines show poor semantic fidelity at that rate.
Takeaways & Limitations
Decoupled foundation-model-guided select-and-generate provides a basis for robust semantic communication at ultra-low bitrates.
Abstract
from arXiv · showhide
Semantic and goal-oriented communication is increasingly studied for 6G, but generalization beyond seen data remains a key weakness under tight rate budgets. Many existing systems overfit their training data and degrade sharply at very low bit rates because they attempt to compress the entire signal. We introduce Foundation Model-Guided Semantic and Goal-Oriented Communication (FMSGOC), a framework that uses broad visual-linguistic Foundation Model priors to mitigate overfitting. It further improves rate efficiency by concentrating bits on sparse, goal-aligned anchors and relying on generative foundation-model priors to reconstruct the masked regions. By decoupling what to send from how to reconstruct, a vision-language foundation model selects and transmits a sparse set of semantic anchors, while a pretrained diffusion model, fine-tuned for masked completion, reconstructs the image at the receiver. In our experiments, FMSGOC reaches 0.039 bits per pixel (BPP), maintains high semantic fidelity (cosine similarity 0.87-0.90 on CIFAR-10), remains robust on previously unseen inputs (0.83-0.86 on ImageNet), and shows good perceptual similarity (0.1278/0.1558, CIFAR-10/ImageNet), outperforming strong end-to-end baselines at lower bit rates.
I. Introduction
FMSGOC addresses poor generalization and sharp low-rate degradation in coupled end-to-end systems by decoupling semantic selection from generative reconstruction. It uses foundation-model priors to transmit sparse, goal-aligned anchors and reconstruct unseen content at ultra-low rates.
- Motivation: End-to-end systems compress entire inputs, coupling semantic extraction, compression, and reconstruction under strict rate budgets.This forces representations to include both important semantics and unimportant data.
- Motivation: Such systems generalize poorly to unseen content or new objectives and degrade sharply at ultra-low bitrates when critical semantics cannot be retained.
- Proposed approach: FMSGOC decouples goal-driven semantic selection from generative reconstruction using complementary foundation models.A VLM selects sparse semantic anchors, while a LoRA fine-tuned diffusion model reconstructs from them.
- Proposed approach: The VLM identifies sparse latent anchors for goal-oriented rate allocation instead of transmitting the whole scene.
- Results: FMSGOC demonstrates competitive rate-distortion-semantic performance at ultra-low bitrates and generalizes to unseen data on CIFAR-10 and ImageNet.
III. System Design
The system encodes an image into a VAE latent grid, uses a VLM-derived importance map to select Top-K goal-aligned anchors, and transmits only those anchors and their indices. The receiver places the anchors into a masked grid and completes the remaining regions generatively.
- System architecture: FMSGOC separates the pipeline into a VLM-guided semantic extractor, a VAE latent encoder, and a LoRA-adapted generative decoder.
- Transmitter: The VAE maps the source image from pixel space to a compact latent grid before semantic selection.
- Transmitter: The VLM generates a goal-aligned importance map used to select the Top-K semantic anchors from the latent grid.The goal-driven mode uses spatial relevance between visual patches and a text goal embedding.
- Channel: Only the sparse latent anchors and their corresponding grid indices form the transmitted channel payload.
- Receiver: At the receiver, transmitted indices place anchors into a masked latent grid while a LoRA-adapted diffusion model fills the unknown regions.
- Anchor selection: Top-K selection retains the full latent feature vector at each of the K highest-scoring spatial cells.
B. Generative Decoder
The generative decoder adapts a frozen diffusion backbone for masked-latent completion and preserves transmitted anchors throughout denoising. The refined latent grid is then decoded into the reconstructed image.
- Decoder role: The receiver reconstructs a coherent image from sparse anchor constraints using an Anchor Preservation Triad to limit semantic drift.
- LoRA adaptation: LoRA adapts the frozen diffusion backbone for masked-latent completion because an unadapted model may overwrite known anchor cells.
- Anchor preservation: Anchor consistency constrains decoder outputs to match received anchor values at known masked locations.
- Anchor preservation: Hard reinjection restores received anchors after every denoising step, preventing semantic drift during inference.
- Image rendering: The refined latent grid is passed through the pretrained VAE decoder to render the reconstructed image.
C. R-D-S Metrics: Rate, Distortion, and Semantics
FMSGOC evaluates communication across rate, perceptual distortion, and semantic alignment, with coded BPP accounting for payload and LDPC overhead. Its sparse Top-K anchor design limits transmission to selected latent cells while retaining quantized latent information.
- Rate, distortion, and semantics are the three evaluation axes, using source image X, reconstruction X̂, and image dimensions H_X × W_X.
- Source BPP R_S measures total payload bits normalized by image size, while reported BPP includes LDPC coding at r_c = 2/3.
- The encoder selects K Top-K anchors from an H_z × W_z latent grid containing N = H_zW_z cells.
- Independent addressing uses a conservative index cost R_idx, while transmitting only selected latent cells keeps the overall BPP ultra-low.
- Selected latent vectors are represented with 8-bit adaptive uniform quantization and a discretized Gaussian likelihood estimate.
A 24-bit frame-level cyclic redundancy check (CRC) is included as 𝐿CRC.
The section defines perceptual distortion with LPIPS and semantic alignment through cosine similarity in a frozen CLIP embedding space.
- LPIPS with a pretrained AlexNet backbone measures per-image perceptual distortion between the source and reconstruction.
- Semantic alignment is measured as cosine similarity between embeddings produced by the frozen CLIP image encoder.
D. Training Objective: Composite Loss for Masked Completion
FMSGOC fine-tunes LoRA modules with a composite objective combining rate, perceptual distortion, semantic alignment, and anchor consistency. The loss weights prioritize the primary R-D-S trade-off while preserving transmitted constraints.
- LoRA fine-tuning uses a composite loss driven primarily by the Rate-Distortion-Semantic objective and penalizes deviations on masked cells.
- The final objective combines the three primary R-D-S terms with an anchor-consistency stabilizer.
- The empirically selected weights are λ_LPIPS = 0.6, λ_sem = 0.2, λ_rate = 0.2, and λ_anchor = 0.1.The auxiliary anchor term is intended to enforce constraint awareness without overpowering the primary generative optimization.
A. Experimental Setup
Experiments train on CIFAR-10 and evaluate on CIFAR-10 and ImageNet, using foundation-model components and comparisons with JPEG and representative E2E neural systems. All methods are evaluated at SNR = 10 dB with matched coded-BPP accounting and native rate controls.
- Dataset and implementation: Training uses CIFAR-10, while evaluation covers both CIFAR-10 and ImageNet to assess generalization.
- Dataset and implementation: The goal-driven selector uses pretrained CLIP ViT-B/32 as its semantic foundation model.
- Dataset and implementation: Stable Cascade supplies a frozen Stage-A VAE and a Stage-B diffusion U-Net adapted for masked completion with LoRA.
- Baselines and operating points: FMSGOC is compared with JPEG, DeepJSCC, and WITT using public checkpoints or the Pillow encoder without additional baseline fine-tuning.
- Baselines and operating points: At SNR = 10 dB, methods share coded-BPP accounting, while operating points use each method’s native rate-control mechanism.FMSGOC varies retained top-K anchor fractions; JPEG varies DCT quantization, and the neural baselines vary channel counts.
C. R-D-S Performance and Generalization
FMSGOC maintains semantic fidelity at ultra-low bitrates and generalizes better than the evaluated baselines on unseen ImageNet data. Its perceptual quality remains stable across datasets, although in-distribution baselines can achieve lower LPIPS at higher cost.
- Rate-semantic performance: 0.039 BPP preserves FMSGOC’s CIFAR-10 CLIP similarity at approximately 0.87-0.90 across the tested low-rate regime.The evaluation uses CLIP cosine similarity, with higher values indicating better semantic alignment.
- Generalization: 0.83-0.86 ImageNet CLIP similarity demonstrates FMSGOC’s semantic generalization under out-of-distribution evaluation.The ImageNet results are attributed to prior knowledge from the foundation-model components.
- Baseline comparison: Approximately 0.60-0.75 ImageNet CLIP similarity is reported for CIFAR-10-trained E2E baselines, while JPEG degrades sharply as bitrate decreases.The E2E baselines are not shown at low CIFAR-10 rates and perform poorly on ImageNet; JPEG is non-semantic.
- Perceptual quality: 0.1278 CIFAR-10 LPIPS and 0.1558 ImageNet LPIPS are FMSGOC’s minimum values at 0.039 BPP.LPIPS is evaluated with lower values indicating better perceptual quality.
- Perceptual quality: CIFAR-10 baselines attain lower minimum LPIPS than FMSGOC, but E2E perceptual quality deteriorates on ImageNet with LPIPS scores of at least 0.6217.The comparison shows a trade-off between in-distribution perceptual quality and out-of-distribution generalization.
D. Ablation Study
The ablation study evaluates FMSGOC against variants without LoRA adaptation or hard reinjection, showing that both components are critical to its semantic fidelity.
- Ablation setup: The study compares the full FMSGOC model with variants without LoRA adaptation and without hard reinjection.Both ablations test whether task-specific adaptation and anchor preservation are necessary.
- Ablation results: The Full Model achieves approximately 0.87-0.90 CLIP similarity and significantly outperforms both ablated variants.The comparison uses CLIP cosine similarity versus BPP.
- Ablation results: Without LoRA, semantic similarity drops to approximately 0.84, indicating that adaptation supports the masked completion task.The ablation removes fine-tuning from the original frozen U-Net.
- Ablation results: Without reinjection, semantic similarity drops to approximately 0.81, indicating that hard reinjection helps prevent drift from transmitted anchors.This variant omits hard reinjection during inference.
- Interpretation: Together, the results identify LoRA adaptation and hard reinjection as complementary components of the proposed design.The study attributes masked-completion capability to LoRA and anchor preservation during inference to hard reinjection.