Source-linked AI summary
Cooperative Multi-Task Semantic Communication for Joint Classification and Regression Tasks
Ahmad Halimi Razlighi, Mohammad Siddiqur Rahman, Maximilian H. V. Tillmann, Edgar Beck, Armin Dekorsy
TL;DR
Existing CMT-SemCom evaluations were limited to homogeneous classification on simple datasets, despite practical perception systems combining classification and regression. This paper extends CMT-SemCom with InfoMax for mixed discrete and continuous variables, evaluates it on Cityscapes, and reports better performance than the considered benchmarks while studying CU capacity.
Problem
CMT-SemCom had been evaluated only on homogeneous classification tasks and relatively simple datasets, leaving heterogeneous classification-regression communication insufficiently evaluated.
Method
The paper extends CMT-SemCom with an InfoMax formulation that jointly models discrete and continuous semantic variables for classification and regression.
Results
CMT-SemCom outperforms single-task SemCom, single-encoder multi-decoder SemCom, and task-agnostic digital transmission while using the same communication resources.
Takeaways & Limitations
The CU/SU feature-extraction split is crucial for balancing information sharing and task specialization, providing practical design insights.
Abstract
from arXiv · showhide
Multi-Task semantic communication (SemCom) prioritizes simultaneous execution of multiple tasks over bit-accurate reconstruction in future intelligent networks. In our prior work [1], we introduced the cooperative multi-task SemCom (CMT-SemCom) framework, in which the semantic encoder is divided into a common unit (CU) and multiple specific units (SUs) to facilitate cooperative multi-task processing. However, CMT-SemCom has been evaluated on homogeneous classification tasks on simplistic datasets, limiting its applicability to real-world perception systems. In this paper, we extend our CMT-SemCom to jointly handle heterogeneous classification and regression tasks on the complex Cityscapes dataset. We adopt the information maximization (InfoMax) principle so that it accommodates mixed discrete and continuous semantic variables. In particular, we benchmark the proposed framework against independent single-task training, a conventional task-agnostic digital transmission, and single-encoder multi-decoder SemCom. Additionally, we investigate the impact of CU capacity on joint task performance, providing design insights. Extensive evaluations demonstrate that CMT-SemCom significantly outperforms the benchmarks.
I. INTRODUCTION
The paper extends cooperative multi-task semantic communication to heterogeneous classification and regression on Cityscapes. It benchmarks this framework against several baselines and studies how shared-unit capacity affects performance.
- Previous evaluations used homogeneous classification tasks on relatively simple datasets, leaving applicability to realistic heterogeneous perception workloads limited.Autonomous-driving perception combines discrete semantic labels with continuous geometric information.
- Existing heterogeneous SemCom methods optimize task-specific losses directly, without an explicit information-theoretic model of semantic variables and their probabilistic relationships.
- The proposed framework is evaluated on Cityscapes to represent realistic urban perception scenarios.
- CMT-SemCom is generalized to jointly process discrete classification and continuous regression variables within one framework.The extension targets heterogeneous tasks rather than only homogeneous image classification.
- The study benchmarks CMT-SemCom against single-task training, task-agnostic digital transmission, and single-encoder multi-decoder SemCom.
- CU capacity is investigated as a trade-off between shared representation richness and task-specific interference.The analysis provides design guidance for selecting CU dimensionality.
II. PROBLEM FORMULATION
The formulation models a multi-task wireless system in which a transmitter extracts task-specific semantics from an image and communicates them to multiple receivers. It represents semantic variables probabilistically and allows discrete or continuous alphabets.
- II. PROBLEM FORMULATION: The formulation establishes a probabilistic model for semantic sources and communication, then defines the joint classification-regression optimization problem.
- A. System Model: A single transmitter observes an H×W×C image and extracts information for N downstream tasks using a split encoder.
- A. System Model: Each receiver decodes a reconstruction of its task’s semantic variable from the received task-specific information.
- A. System Model: Semantic variables may use continuous alphabets with Z_i = R or discrete class alphabets.
- A. System Model: The common unit preprocesses observations for related tasks, while each specific unit converts the shared output into a length-d codeword.
- A. System Model: Each codeword traverses an AWGN channel, and receiver i observes y_i = x_i + n_i before inferring the reconstructed semantic variable.
B. Problem Statement
The framework designs the CU, SUs, and decoders by maximizing mutual information between each channel output and its corresponding semantic variable. Neural variational approximations and Monte Carlo sampling make the objective trainable for mixed task types.
- B. Problem Statement: InfoMax maximizes mutual information between each channel output Y_i and its corresponding semantic variable Z_i.
- B. Problem Statement: The CU and SU conditional distributions, p(c|s) and p(x_i|c), are the optimization variables for the semantic encoders.
- B. Problem Statement: The mutual-information objective is expanded into an approximated optimization function for neural-network training.
- B. Problem Statement: Neural networks parameterize the CU, SUs, and decoders, including q_ψi(z_i|y_i) as an approximation to the decoder posterior.
- B. Problem Statement: The encoder combines semantic and channel coding through the learned conditional distribution linking semantic observations to transmitted signals and channel outputs.
- B. Problem Statement: Monte Carlo sampling over data and noise produces empirical loss functions for classification and regression tasks.
A. Classification Loss Function
For classification tasks, the InfoMax objective is estimated with sampled data and channel noise, then implemented by minimizing the corresponding sample-estimated cross-entropy. Inference selects the maximum-likelihood semantic output.
- A. Classification Loss Function: The classification objective estimates negative cross-entropy using data samples and noise samples.
- A. Classification Loss Function: The encoder output sample size and channel sample y^{m,l} = x^m + n^l are fixed equally for each batch during implementation.
- A. Classification Loss Function: Training minimizes the sample-estimated cross-entropy instead of maximizing its negative.
- A. Classification Loss Function: Classification inference chooses the maximum-likelihood output under the learned decoder distribution.
B. Regression Loss Function
The regression loss models each continuous semantic variable with a pixel-wise independent Gaussian distribution, whose mean is predicted by a neural network. This assumption makes the likelihood tractable and yields an empirical regression loss.
- B. Regression Loss Function: A pixel-wise independent Gaussian distribution models each continuous semantic variable, with fixed variance and a mean vector predicted from the decoder input.The mean vector has one value for each of the K = H × W pixels.
- B. Regression Loss Function: The Gaussian probability density is fully determined by its mean when the variance is fixed, so the neural network outputs µ(yi).
- B. Regression Loss Function: The Gaussian log-likelihood is integrated into the InfoMax objective to obtain an empirical regression loss based on squared prediction errors.The likelihood expression omits terms constant with respect to the maximization.
- B. Regression Loss Function: Maximum-likelihood estimation selects the semantic-variable estimate at the Gaussian density's maximum, which occurs at the predicted mean.
IV. SIMULATION RESULTS
The simulation-results section evaluates CMT-SemCom against baseline task-execution methods and studies how CU capacity affects performance. These experiments address both comparative effectiveness and the shared-versus-specific processing split.
- IV. SIMULATION RESULTS: The section evaluates CMT-SemCom task execution by comparing it against baseline methods.
- IV. SIMULATION RESULTS: The experiments investigate how CU capacity affects the performance of the cooperative multi-task architecture.
- IV. SIMULATION RESULTS: Together, the evaluations cover both benchmark comparison and the architectural role of the CU.
A. Simulation Setup
The simulation setup uses fine-annotated Cityscapes images for three tasks: two semantic-segmentation classifications and pixel-level depth regression. The proposed system is evaluated with the dataset's image and label structure.
- A. Simulation Setup: The experiments use 5,000 fine-annotated Cityscapes urban street-scene images containing semantic labels and stereo disparity maps.Disparity maps are converted into normalized floating-point depth labels.
- A. Simulation Setup: The input images have resolution 256 × 512 × 3, and the task formulation includes a total loss for the three-task system.
- A. Simulation Setup: CMT-SemCom performs three tasks: five-class semantic segmentation, binary semantic segmentation, and pixel-level depth estimation.The first two tasks are classification tasks, while the third is regression.
2) CMT-SemCom NN Architecture:
The proposed architecture divides processing between a shared CU and task-specific SUs while keeping total per-task depth fixed. It uses task-specific decoders and compares against baselines under equal trainable-parameter budgets.
- 2) CMT-SemCom NN Architecture:: The CU contains one convolutional layer and F ResNet blocks, while each SU contains the remaining 9 − F blocks.Varying F tests how shared feature extraction and task-specific refinement affect performance.
- 2) CMT-SemCom NN Architecture:: All task decoders share the same architecture except for their final task-specific output layers, progressively upsampling received latent representations to image resolution.
- 2) CMT-SemCom NN Architecture:: CMT-SemCom is compared with independent single-task SemCom, single-encoder multi-decoder SemCom, and task-agnostic digital transmission using equal trainable-parameter counts.The baselines use the same general neural-network components for fair comparison.
4) Evaluation Metrics:
Segmentation is evaluated with pixel-level classification accuracy and IoU, while depth regression uses MSE and δ1 accuracy.
- Pixel-level classification accuracy measures the fraction of valid pixels whose inferred labels match ground truth.
- IoU measures segmentation overlap by dividing the intersection by the union of true positives, false positives, and false negatives.
- Task 2 uses IoU for building segmentation, whereas Task 1 uses mean IoU averaged across its defined classes.
- Depth estimation is evaluated using MSE and δ1 accuracy, which measures the fraction of pixels whose ground-truth-to-prediction depth ratio is below 1.25.
B. Evaluations
The evaluation tests CU/SU capacity allocation and compares CMT-SemCom with single-task, multi-decoder, and digital baselines across SNR. CMT-SemCom performs best overall, including under bandwidth-constrained, low-SNR conditions.
- Architectural Investigations: A balanced CU/SU split, CMT-CU5/SU4, achieves the best performance for all three tasks and becomes the default architecture.
- Benchmark Comparisons: From −5 to 10 dB SNR, performance improves across all metrics for every considered method because channel conditions become more favorable.
- Benchmark Comparisons: CMT-SemCom consistently outperforms SEMD-SemCom on Tasks 2 and 3, while achieving slightly higher Task 1 IoU despite equal pixel-level accuracy.
- Benchmark Comparisons: CMT-SemCom outperforms ST-SemCom on all three tasks across every evaluated SNR value.
- Benchmark Comparisons: CMT-SemCom achieves better task execution accuracy than TAD, especially with limited bandwidth and at low SNR.
- Benchmark Comparisons: On an illustrative Cityscapes example, CMT-SemCom produces less fragmented segmentation masks and larger low-error depth regions than ST-SemCom.
V. CONCLUSION
The extended CMT-SemCom framework jointly supports classification and regression on Cityscapes using InfoMax for mixed discrete and continuous semantic variables, outperforming several benchmarks under equal communication resources.
- Under the same communication resources, the proposed framework outperforms single-task SemCom, single-encoder multi-decoder SemCom, and task-agnostic digital transmission.
- InfoMax extends CMT-SemCom to heterogeneous perception by accommodating mixed discrete and continuous semantic variables.
- The distribution of feature extraction between the shared CU and task-specific SUs is crucial for balancing information sharing and task specialization.