PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic-preserved Communication System for Highly Efficient Speech Transmission

Tianxiao Han, Qianqian Yang, Zhiguo Shi, Shibo He, Zhaoyang Zhang

arXiv:2205.12727v1eess.AScs.SD

TL;DR

The paper addresses efficient speech transmission by sending semantic-relevant information for recognition and a compact speech-related supplement for reconstruction. It proposes an end-to-end deep-learning transceiver with subword-based semantic decoding and CTC alignment, using 16% and 0.2% of existing symbol amounts for speech-to-text and speech-to-speech transmission, respectively.

  • Problem

    Traditional communication transmits abstract symbols, motivating a speech system that sends semantic-relevant information efficiently for recognition and reconstruction.

  • Method

    An end-to-end deep-learning transceiver extracts semantic information from speech spectrums, decodes subword transcriptions, and adds CTC-aligned duration, pitch, and power information for speech reconstruction.

  • Results

    16% of existing transmitted symbols are required for speech-to-text transmission, while speech-to-speech transmission requires 0.2%.

  • Takeaways & Limitations

    The proposed approach improves transmission efficiency while outperforming existing methods in predicted-text accuracy and recovered-speech quality.

Abstract

from arXiv · show

Deep learning (DL) based semantic communication methods have been explored for the efficient transmission of images, text, and speech in recent years. In contrast to traditional wireless communication methods that focus on the transmission of abstract symbols, semantic communication approaches attempt to achieve better transmission efficiency by only sending the semantic-related information of the source data. In this paper, we consider semantic-oriented speech transmission which transmits only the semantic-relevant information over the channel for the speech recognition task, and a compact additional set of semantic-irrelevant information for the speech reconstruction task. We propose a novel end-to-end DL-based transceiver which extracts and encodes the semantic information from the input speech spectrums at the transmitter and outputs the corresponding transcriptions from the decoded semantic information at the receiver. For the speech to speech transmission, we further include a CTC alignment module that extracts a small number of additional semantic-irrelevant but speech-related information for the better reconstruction of the original speech signals at the receiver. The simulation results confirm that our proposed method outperforms current methods in terms of the accuracy of the predicted text for the speech to text transmission and the quality of the recovered speech signals for the speech to speech transmission, and significantly improves transmission efficiency. More specifically, the proposed method only sends 16% of the amount of the transmitted symbols required by the existing methods while achieving about 10% reduction in WER for the speech to text transmission. For the speech to speech transmission, it results in an even more remarkable improvement in terms of transmission efficiency with only 0.2% of the amount of the transmitted symbols required by the existing method.

I. INTRODUCTION

Semantic communication targets efficient transmission by sending meaning-related information rather than abstract symbols. This paper applies that approach to speech recognition and reconstruction with a compact end-to-end deep-learning transceiver.

  • Proposed system: The proposed system extracts text-related semantic features from speech and removes repeated or semantically irrelevant features before transmission.An attention-based alignment module matches semantic features to text content, while redundancy removal drops unnecessary information.
  • Proposed system: The approach combines beam-search decoding and a pretrained-language-model semantic corrector to improve transcription prediction.The corrector uses learned semantic knowledge to address semantic errors in predicted subword sequences.
  • Proposed system: A compact additional speech-information extractor provides duration, pitch, and power information for speech reconstruction.A CTC alignment-based extractor and speech reconstructor support recovery of the speech signal at the receiver.
  • Proposed system: Subword tokens preserve semantic meaning, avoid unseen-word problems, and improve predicted transcription accuracy compared with character- or word-based tokens.Any new word can be represented as a set of subwords, avoiding out-of-vocabulary words.

II. SYSTEM MODEL

The system model targets compact semantic transmission of speech for two receiver tasks: recovering the corresponding text or recovering the speech signal. Performance is evaluated separately for speech-to-text and speech-to-speech transmission.

  • System objectives: The system sends compact semantic information from input speech over the channel.Its stated goal is highly efficient speech transmission.
  • Transmission tasks: Speech-to-text transmission recovers the corresponding text transcription at the receiver.This is one of the two defined transmission tasks.
  • Transmission tasks: Speech-to-speech transmission recovers the speech signal at the receiver.The paper introduces separate performance metrics for the two transmission tasks.

A. Speech Transmission System

The speech transmission system encodes semantic and compact speech-related information for channel transmission, then decodes text and reconstructs speech at the receiver. It transmits meaning-related information for both tasks while preserving reconstruction quality.

  • Transmitter: The transmitter derives latent semantic representation L and intermediate representation H from the input speech spectrum S.The semantic encoder produces these representations before channel encoding.
  • Transmitter: The additional speech-information extractor computes duration, pitch, and power information D from H and S.These features supplement semantic information for speech reconstruction.
  • Channel transmission: The channel encoder maps L and D into transmitted symbols X over the physical channel.The receiver obtains channel outputs that are decoded back into semantic and speech-related representations.
  • Receiver: The semantic decoder converts decoded semantic representation bL into predicted transcription bG with help from a semantic corrector.The corrector retains semantic information from previous context.
  • Receiver: The semantic reconstructor uses predicted transcription bG and decoded speech information bD to recover a speech spectrum, followed by a pretrained GAN vocoder.The vocoder recovers the desired speech signals from the reconstructed spectrum.
  • Transmission principle: Only meaning-related information plus a compact set of speech-related features is transmitted for speech-to-speech communication.Experiments report improved transmission efficiency while preserving transcription and speech reconstruction qualities.

B. Performance Metrics

The paper evaluates speech-to-text transmission with WER and semantic similarity, and speech-to-speech reconstruction with MCD and MOS. These metrics assess transcription accuracy, sentence-level semantic correspondence, spectral distortion, and perceived naturalness.

  • Speech-to-text metrics: WER measures speech-to-text transcription errors using substitutions, deletions, insertions, and the reference word count.The metric is defined from S, D, I, and N.
  • Speech-to-text metrics: Semantic similarity scores quantify sentence similarity between predicted and original transcriptions on a 0-to-1 scale.A score of 1 indicates semantic equivalence, while 0 indicates no relevance.
  • Speech-to-speech metrics: MCD evaluates speech-spectrum reconstruction quality, with smaller values indicating closer agreement with the original spectrum.The spectra are aligned using Dynamic Time Warping before calculating MCD.

A. Speech to Text Transmission

The speech-to-text subsystem extracts semantic information from speech spectrums, compresses it by removing redundancy, and reconstructs transcriptions at the receiver. Its soft alignment concentrates resources on semantically significant speech portions.

  • Semantic encoder: The semantic encoder extracts transcription-related representations from speech spectrums and removes semantic-irrelevant redundancy before transmission.It uses VGG, BLSTM, fully connected, soft alignment, and redundancy removal components.
  • Semantic encoder: The soft alignment module uses attention and LSTM processing to align speech features with semantic text and produce latent representations.Attention scores are computed from query, key, and feedback information.
  • Semantic encoder: q is always smaller than 10 percent of N, concentrating transmission resources on semantically significant parts.The paper attributes this efficiency to the derived attention scores.
  • Redundancy removal: The redundancy removal module cuts sequences after EOS and removes special tokens, saving about 59.4% and approximately 4.5% of transmission length, respectively.Subwords are used as tokens, reducing the vocabulary from 15003 to 1001.

2) Channel Encoder and Channel Decoder:

The channel encoder and decoder learn an end-to-end mapping between compact semantic representations and transmitted symbols. The receiver then decodes semantic features for transcription, using beam search and an external language-model-based semantic corrector.

  • Channel encoder and decoder: Two cascaded fully connected layers map compact semantic representations to complex-valued transmitted symbol sequences.The real and imaginary signal components occupy separate channels.
  • Channel encoder and decoder: The channel decoder reshapes received symbols and applies fully connected layers to recover text-related semantic features for transcription decoding.The learned channel encoder and decoder adapt neural-network weights through end-to-end optimization to mitigate channel noise.
  • Channel encoder and decoder: Unlike conventional channel coding, the proposed channel encoder reduces transmit-symbol dimensions instead of introducing redundancy.This design is presented as a response to the semantic-communication objective.
  • Semantic decoding: Beam search retains k partial transcriptions and selects the most likely extensions at each iteration.The vocabulary contains 1001 tokens, and the decoder outputs the most likely transcription in its beam.
  • Semantic decoding: A pretrained RNN language model supplies external semantic knowledge to correct errors introduced by noisy transmission.Its distribution is combined with the beam-search distribution using a specific weight.

B. Speech to Speech Transmission

The speech-to-speech subsystem augments semantic transmission with compact speech-related information and reconstructs speech from received text, duration, pitch, and power information. A CTC-based alignment module extracts phoneme timing for this reconstruction.

  • System extension: The speech-to-speech subsystem adds a transmitter-side speech-information extractor and receiver-side speech reconstructor to the speech-to-text system.The additional information includes duration, pitch, and power.
  • Additional speech information extractor: The CTC alignment module uses frame-level predictions and Viterbi decoding to estimate the duration of each predicted phoneme.A pretrained subword-to-phoneme model supplies corresponding phonemes.
  • Additional speech information extractor: The additional speech information D contains duration, pitch, and power features and is described as negligible compared with semantic representation L.It is transmitted using traditional channel coding and can be recovered losslessly.
  • Speech reconstructor: The speech reconstructor combines text embeddings with transmitted or predicted duration, pitch, and power information to recover speech spectrums.A transformer-based TTS decoder produces spectrums, and HiFiGAN generates the final speech signals.
  • Speech reconstructor: The reconstructed latent representation integrates semantic-relevant text information with speech-related information before TTS decoding.The TTS decoder contains six transformer layers.

C. Model Training and Testing

The proposed semantic communication network is trained in two stages, first learning semantic transcription and auxiliary speech information, then optimizing transmission and reconstruction through the physical channel.

  • Data preparation: Training data are expanded by changing speech speed and adding simulated environmental noise with SNR values from 0dB to 15dB.Speech is sped up to 1.1× or slowed to 0.95×, and room-impulse-response noises are added.
  • Training stages: Two-stage training first jointly trains the semantic encoder, semantic decoder, and additional speech information extractor while bypassing the noisy channel.The first stage uses spectrum sequences and transcriptions, with cross-entropy and CTC objectives.
  • Training stages: Teacher forcing accelerates convergence in the soft alignment module by initially replacing incorrect fully connected-layer outputs with true transcriptions.This strategy addresses the feedback loop during early training.
  • Training stages: The second stage freezes the semantic encoder and additional-information extractor, then trains the channel encoder, channel decoder, and semantic decoder under random-SNR physical channels.For speech-to-speech transmission, mean square error between input and recovered spectrums is also minimized.
  • Testing: Testing uses separate speech sequences from training, and the trained models produce transcriptions or reconstruct speech through the corresponding testing pipelines.The algorithms generate spectra, encode semantic information, transmit through the channel, decode, and evaluate the outputs.

IV. EXPERIMENT AND NUMERICAL RESULTS

Experiments compare the proposed approaches with deep-learning speech transmission benchmarks under AWGN and Rayleigh channels using the Librispeech dataset and shared data settings.

  • Experimental setup: Experiments evaluate speech transmission under AWGN and Rayleigh channels using the Librispeech dataset.The dataset provides 960 hours for speech-to-text work, with separate training and testing utterances.
  • Compared approaches: Speech-to-text comparisons include DeepSC-SR, text-oriented DeepSC combined with the semantic encoder, and the proposed benchmarks.The proposed variants include SE-DeepSC and DeepSC-S-SR.
  • Experimental setup: The compared methods use the same input-spectrum dimensions, training data augmentation strategies, datasets, and channel-SNR conditions for fairness.Training SNR varies randomly from 5dB to 10dB for speech-to-text transmission.
  • Compared approaches: Speech-to-speech comparisons include DeepSC-S and variants using or omitting the proposed semantic encoder.A baseline using the proposed semantic encoder and decoder without channel components provides an upper-bound performance reference.
  • Evaluation: Speech-to-text testing computes word error rate and sentence similarity after decoding predicted transcriptions across SNR values.The testing procedure transmits encoded semantic representations through fading channels before evaluation.

A. Performance comparison of the speech to text transmission

The proposed speech-to-text system outperforms the compared approaches across channel conditions in WER and sentence similarity, with semantic correction contributing to the improvement.

  • Overall comparison: The proposed method significantly outperforms other methods under both channel conditions in both WER and sentence similarity.Its performance remains steady and closer to the baseline across SNR values.
  • Overall comparison: The proposed approach achieves around 15% WER instead of the 40% reported for DeepSC-SR.This comparison is reported for the speech-to-text evaluation shown in Fig. 6.
  • Overall comparison: SE-DeepSC performs worse than the other compared approaches, particularly under low-SNR conditions.The paper suggests semantic errors in its output may contribute to this result.
  • Performance factors: Subwords, data augmentation, beam search, and semantic correction are identified as factors associated with the improvement over the previous approach.The semantic corrector uses internal and external semantic knowledge to fix semantic errors.
  • Performance factors: The semantic corrector creates a clear performance gap over the baseline without correction, while greedy semantic decoding performs worse than the proposed methods.Beam search avoids some greedy-decoder word errors, and semantic correction fixes additional mistakes toward the ground truth.

B. Transmission Efficiency

Redundancy removal sharply reduces transmitted speech information while preserving semantic content, and compact additional speech information improves reconstruction quality with limited transmission cost.

  • Speech-to-text efficiency: Speech-spectrum frames substantially outnumber transcription tokens, indicating that many frames are semantically irrelevant for transmission.The redundancy-removal module shortens the transmitted representation while preserving semantic information.
  • Speech reconstruction: The proposed system achieves MOS results comparable to DeepSC-S, while its MOS without additional speech information is better than with that information.The paper attributes this difference to MOS measuring naturalness rather than closeness to the original speech.
  • Speech reconstruction: Additional speech information significantly improves recovered-spectrum quality according to MCD, which measures distance from the original spectrum.Without it, predicted speech can have less similar pitch and power and a longer duration than the original.
  • Speech reconstruction: DeepSC-S obtains lower MCD because it targets exact spectrum recovery, whereas the proposed method removes semantic-irrelevant factors such as environmental noise.The comparison reflects different reconstruction objectives rather than identical spectrum targets.
  • Speech-to-speech efficiency: 1120 transmitted symbols per sentence are used for speech-to-speech transmission, less than 0.2% of DeepSC-S’s 655360 symbols.The additional speech information has dimension 64 and is sent once per utterance.

V. CONCLUSION

The proposed semantic-oriented speech communication system separates semantic information from compact speech-related information to support both recognition and reconstruction. It improves transmission efficiency while maintaining or improving task performance over existing methods.

  • V. CONCLUSION: The system transmits semantic-relevant information for speech recognition and a compact additional set of speech-related information for speech reconstruction.The speech-to-speech pipeline uses CTC alignment to extract additional information that supports signal recovery.
  • V. CONCLUSION: Attention-based soft alignment and redundancy removal extract text-related semantic features while dropping semantically redundant content.The numerical results show that this process reduces 90% of the latent semantic features produced by the semantic encoder.
  • V. CONCLUSION: A beam search semantic decoder exploits long-time dependencies to identify likely transcriptions, while a pretrained language model further corrects predicted text using semantic knowledge.The speech reconstructor combines received semantic information and additional speech-related information with text-to-speech decoding and a pretrained GAN.
  • V. CONCLUSION: The proposed method improves predicted-text accuracy for speech-to-text transmission and achieves comparable recovered-speech quality to existing approaches for speech-to-speech transmission.The conclusion reports improvements over previous and existing methods in these task-specific evaluations.
  • V. CONCLUSION: 16% of the transmitted symbols required by the state-of-the-art approach are used for speech-to-text transmission, while speech-to-speech transmission uses only 0.2% of the existing method's amount.These reductions are reported alongside improved transmission efficiency over existing methods.
Loading 2205.12727v1…