PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning Enabled Semantic Communications with Speech Recognition and Synthesis

Zhenzi Weng, Zhijin Qin, Xiaoming Tao, Chengkang Pan, Guangyi Liu, Geoffrey Ye Li

arXiv:2205.04603v2eess.AScs.SD

TL;DR

The paper addresses efficient speech transmission for recognition and synthesis under constrained and varying channel conditions. DeepSC-ST transmits text-related semantic features, recovers text, and synthesizes speech using speaker information; simulations report advantages over conventional and existing semantic systems, especially at low SNR.

  • Problem

    Conventional communication systems face high transmission demands and scarce spectrum resources, while users may require only task-related semantic information from speech.

  • Method

    DeepSC-ST jointly extracts and channel-encodes low-dimensional text-related speech features, recovers text at the receiver, and synthesizes speech from the text and speaker information.

  • Results

    DeepSC-ST outperforms conventional and existing semantic communication systems, particularly in low-SNR regimes and across tested channel conditions.

  • Takeaways & Limitations

    The system supports speech recognition and synthesized speech output while reducing the amount of transmitted speech information for users requesting text.

Abstract

from arXiv · show

In this paper, we develop a deep learning based semantic communication system for speech transmission, named DeepSC-ST. We take the speech recognition and speech synthesis as the transmission tasks of the communication system, respectively. First, the speech recognition-related semantic features are extracted for transmission by a joint semantic-channel encoder and the text is recovered at the receiver based on the received semantic features, which significantly reduces the required amount of data transmission without performance degradation. Then, we perform speech synthesis at the receiver, which dedicates to re-generate the speech signals by feeding the recognized text and the speaker information into a neural network module. To enable the DeepSC-ST adaptive to dynamic channel environments, we identify a robust model to cope with different channel conditions. According to the simulation results, the proposed DeepSC-ST significantly outperforms conventional communication systems and existing DL-enabled communication systems, especially in the low signal-to-noise ratio (SNR) regime. A software demonstration is further developed as a proof-of-concept of the DeepSC-ST.

I. INTRODUCTION

The paper proposes DeepSC-ST, a semantic communication system that transmits speech through task-related text features and supports both text recognition and speech synthesis. It targets higher transmission efficiency by omitting speech characteristics that are unnecessary for users requesting text information.

  • Motivation and system concept: DeepSC-ST omits speaker voice, speech delay, and background noise from transmission, lowering network traffic for users requesting text information.The receiver can nevertheless restore speech using the recognized text and pre-registered speaker information.
  • Contributions: DeepSC-ST uses joint semantic-channel coding for speech-input communication scenarios.The system is designed for speech transmission and extracts task-relevant semantic information for communication.
  • Contributions: Text-related semantic features extracted with CNN- and RNN-based components reduce transmitted data and communication resources without performance degradation.The features focus on information needed for speech recognition rather than the full speech signal.
  • Contributions: The receiver converts recovered semantic features into text and synthesizes speech from the recognized text and speaker information.The synthesis module uses CNN- and RNN-based neural networks, with speaker information available through the user identity.
  • Contributions: A software demonstration accepts real human speech input and produces recognized text and synthesized speech through an operable interface.The demonstration serves as a proof-of-concept for the proposed system.

II. RELATED WORK

The related work spans semantic communication for text, audio, images, and video, alongside neural speech recognition and synthesis. DeepSC-ST combines these lines of research by transmitting text-related speech semantics for recognition and reconstructing speech at the receiver.

  • DL-enabled semantic communications: Prior semantic communication systems address text transmission, audio transmission, image and video reconstruction, and intelligent vision tasks.Examples include transformer-based text systems, audio autoencoders, GAN-based image coding, and semantic video conferencing.
  • Speech recognition: Neural speech recognition evolved from HMM-based systems through DNN-HMM hybrids, temporal models, deep LSTMs, and RNN Transducers.These approaches progressively support character-level transcription and sequence modeling for speech recognition.
  • Speech synthesis: Tacotron and Tacotron 2 simplified neural speech synthesis, while non-autoregressive pipelines target lower generation latency for long sentences.Tacotron 2 generates waveforms from normalized character sequences and produces realistic human voice.
  • System positioning: The considered semantic speech system transmits low-dimensional text-related features, recognizes text, and reconstructs speech using the recognized text and user ID.This system model connects semantic communication with both speech recognition and speech synthesis tasks.

A. Input Spectrum and Text Information

The system converts speech into spectrum and text-related semantic features, transmits encoded symbols over a noisy channel, and reconstructs the text transcription at the receiver.

  • Input Spectrum and Text Information: The input speech sequence is divided into frames and transformed into a normalized spectrum containing its characteristics.Processing uses a Hamming window, FFT, logarithm operation, and normalization.
  • Input Spectrum and Text Information: The semantic encoder converts the input spectrum into text-related semantic features, while the channel encoder maps them into transmitted symbols.The transmitter uses neural networks with parameters α and β, jointly represented as θT = (α, β).
  • Input Spectrum and Text Information: The wireless channel receives encoded symbols x and produces received symbols y, with linear transmission modeled using channel coefficients h and Gaussian noise.The encoded symbols are normalized, and the noise vector has variance σ2 for each channel.
  • Input Spectrum and Text Information: At the receiver, the channel decoder maps received symbols into probability-based text-related semantic features, and the feature decoder produces the text transcription.The probability vectors contain 29 probabilities corresponding to the token set.
  • Input Spectrum and Text Information: Speech recognition is trained to maximize p(t|s) using connectionist temporal classification, which sums probabilities over valid alignments between the text sequence and spectrum.Blank tokens are removed when obtaining the final transcription, and the blank token is the 29th token in the English token set.

D. Speech Synthesis Module

The speech synthesis module converts decoded text-related semantic features into a reconstructed speech sequence using a neural network with parameters χ.

  • Speech Synthesis Module: The speech synthesis module takes the feature decoder output bt and generates speech sample sequence c_m to reconstruct the original sequence m.Its neural network is designed to make the reconstructed waveform as close to the original as possible.
  • Speech Synthesis Module: The reconstructed speech sample sequence is represented as the output of the speech synthesis neural network parameterized by χ.χ(·) denotes the speech synthesis module with respect to parameters χ.

E. Performance Metrics

The paper evaluates recovered-text quality using character- and word-level error rates, while noting their interpretation and limitations.

  • Character error-rate (CER) and word error-rate (WER) quantify recognition accuracy by counting incorrect characters or words in recovered text.These metrics measure substitutions, deletions, and insertions relative to the original transcription.
  • WER = SW + DW + IW, where SW, DW, and IW count word substitutions, deletions, and insertions, respectively.NW is the number of words in the reference text, as specified in the metric definition.
  • CER and WER may exceed one when many deletions occur, and CER is typically lower than WER for the same sentence.
  • A recognized sentence is usually readable when CER is below approximately 0.15.

2) Speech Synthesis Task:

Speech synthesis quality is assessed by comparing synthesized and real speech through feature-distribution distances computed with a pretrained speech recognition model.

  • FDSD and KDSD quantify the distribution similarity between synthesized speech and real speech.The paper adopts unconditional Fréchet deep speech distance and unconditional kernel deep speech distance as evaluation metrics.
  • FDSD compares the means and covariance matrices of features extracted from real and synthesized speech.The features are obtained by passing both speech sequences through a publicly available deep speech recognition model.
  • KDSD evaluates the same real-versus-synthesized speech comparison using a kernel function.The kernel function is denoted by kf(·) in the metric formulation.

IV. SEMANTIC COMMUNICATIONS FOR SPEECH RECOGNITION AND SYNTHESIS

DeepSC-ST transmits text-related speech semantics for recognition and optionally reconstructs speech from recovered text and speaker identity. Its pipeline uses CNN/RNN semantic encoding, dense channel processing, greedy decoding, and Tacotron 2 synthesis.

  • A. Model Description: CNN and RNN modules extract text-related semantic features for speech recognition, while dense layers perform channel encoding and decoding.The recovered features are converted into text transcriptions at the receiver.
  • A. Model Description: The transmitter converts speech spectra into semantic features, maps them to symbols, and sends them through the physical channel.
  • C. Greedy Decoder: The receiver reshapes received symbols, decodes the channel representation, and uses a greedy decoder to produce text transcriptions.At each decoding step, the maximum-probability token is selected and concatenated into the final transcription.
  • A. Model Description: Tacotron 2 reconstructs speech from recovered text and the corresponding user ID, and it can be omitted when users request only text.The synthesis module is trained separately from speech recognition.
  • C. Greedy Decoder: The greedy decoder is not implemented as a neural network because maximum-probability selection and token mapping are non-differentiable.

E. Speech Synthesis Module

The speech synthesis module uses Tacotron 2 to convert recognized text and speaker identity into waveform speech through spectrogram prediction and neural vocoding.

  • E. Speech Synthesis Module: Tacotron 2 reconstructs speech sample sequences from token sequences and user identity.It combines a spectrogram prediction network with a WaveNet vocoder.
  • E. Speech Synthesis Module: The WaveNet vocoder converts predicted mel-frequency spectrograms into time-domain waveforms by predicting waveform-distribution parameters.Its training maximizes speech-waveform log-likelihood with respect to trainable parameters.
  • E. Speech Synthesis Module: The synthesis training procedure initializes parameters, generates spectra, and optimizes the model while CTC loss has not converged.The listed training steps transmit encoded symbols through the channel, compute CTC loss, and update parameters using SGD.
  • E. Speech Synthesis Module: The testing procedure evaluates speech recognition across channel conditions and a wide range of SNR values by adding Gaussian noise before decoding.It outputs recovered text transcriptions after channel transmission and feature decoding.
  • E. Speech Synthesis Module: The spectrogram prediction network encodes token sequences with convolutional and bidirectional LSTM layers, then predicts mel-frequency spectrogram frames sequentially.The decoder uses previous synthesized frames, encoded features, and user ID to predict each current frame.

V. NUMERICAL RESULTS

The experiments compare DeepSC-ST with conventional and existing semantic communication systems across different channels. Speech recognition and synthesis are evaluated on downsampled LJSpeech data under an accurate-CSI receiver assumption.

  • The experiments compare DeepSC-ST with conventional and existing semantic communication systems under different channel conditions.The receiver is assumed to have accurate channel state information (CSI).
  • Both speech recognition and speech synthesis tasks are evaluated using the LJSpeech English-speech dataset.The original sampling rate is 22,050 Hz.
  • The speech data are downsampled from 22,050 Hz to 16,000 Hz for the experiments.
  • The simulation environment uses TensorFlow 2.4.

A. Simulation Setting and Benchmarks

The study compares DeepSC-ST with four speech-transmission benchmarks spanning conventional and DL-enabled semantic communication paradigms.

  • Simulation Setting and Benchmarks: DeepSC-ST uses two CNN modules and six BRNN modules in its semantic encoder.Each CNN has 32 filters, each BRNN has 800 GRU units, and the channel encoder and decoder use dense layers with 40-unit layers.
  • Simulation Setting and Benchmarks: The comparison includes speech, text, and feature transceivers, plus the hybrid SR+DeepSC system.These benchmarks respectively transmit speech signals, text sequences, extracted semantic features, or combine speech recognition with DeepSC.
  • Simulation Setting and Benchmarks: Conventional communication is used by the speech, text, and feature transceivers, whereas SR+DeepSC and DeepSC-ST use DL-enabled semantic communication.

B. Complexity Analysis

DeepSC-ST has complexity comparable to three benchmarks, transmits substantially fewer encoded symbols, and achieves strong recognition performance across channel conditions, especially at low SNR.

  • Complexity Analysis: 85,796,042 trainable parameters place DeepSC-ST at the same complexity level as the speech, text, and feature transceivers.SR+DeepSC has 92,212,016 parameters, a 7.48% increase over DeepSC-ST.
  • Complexity Analysis: Nearly tenfold lower transmission data than the speech and feature transceivers is achieved by DeepSC-ST.The text transceiver and SR+DeepSC each require an average of 60 encoded symbols for the evaluated speech samples.
  • Speech Recognition Experiments: When the learning rate is 0.0001 or 0.00005, CTC loss decreases to around 10 after 20 epochs and converges after about 40 epochs.
  • Speech Recognition Experiments: DeepSC-ST has lower CER scores than all benchmarks across the tested AWGN, Rayleigh, and Rician channel environments.It significantly outperforms benchmarks from -12 dB to 4 dB for AWGN and from -12 dB to 8 dB for Rayleigh and Rician channels.
  • Speech Recognition Experiments: DeepSC-ST provides lower WER across varied channel conditions and outperforms conventional systems particularly when SNR is low.It outperforms the text and feature transceivers when SNR<8 dB.

D. Experiments for Speech Synthesis Task

DeepSC-ST achieves better synthesized-speech recovery than the evaluated benchmarks across tested channel conditions, including low-SNR regimes. A software demonstration exposes the system's speech input, channel, SNR, recognized text, and synthesized speech outputs.

  • FDSD and KDSD scores are computed under the adopted Deep Speech 2 model and may differ with other model parameters or models.
  • DeepSC-ST obtains lower FDSD and KDSD scores than SR+DeepSC under all tested channel conditions.
  • From -8 dB to 2 dB in AWGN, and nearly -10 dB to 10 dB in Rayleigh and Rician channels, DeepSC-ST achieves better speech recovery than the speech, feature, and text transceivers.
  • When AWGN SNR is below around 4 dB, received speech from the speech transceiver is invalid, whereas DeepSC-ST synthesized speech remains acceptable for SNR>-10 dB.
  • The software demonstration lets users choose or record a .wav input, select among three fading channels, specify SNR, and obtain recognized text and synthesized speech.
Loading 2205.04603v2…