Source-linked AI summary
Deep Learning for Joint Source-Channel Coding of Text
Nariman Farsad, Milind Rao, Andrea Goldsmith
TL;DR
The paper addresses whether separate source and channel coding remains suitable for finite-length text transmission under encoding constraints. It develops a neural joint source-channel encoder-decoder that targets semantic preservation, and reports lower word error rates than separate coding at small bit budgets.
Problem
Finite block lengths and encoding constraints remove the guaranteed optimality of separate source and channel coding for noisy text transmission.
Method
A recurrent neural encoder, binarization layer, differentiable channel layer, and recurrent decoder jointly encode and recover sentence information.
Results
The proposed deep learning encoder and decoder outperform separate source and channel coding in word error rate, especially with few bits per sentence.
Takeaways & Limitations
Semantic information can be preserved even when the decoded sentence contains different words, additions, or omissions.
Takeaways & Limitations
The current algorithm uses a fixed bit length for sentences of different lengths.
Abstract
from arXiv · showhide
We consider the problem of joint source and channel coding of structured data such as natural language over a noisy channel. The typical approach to this problem in both theory and practice involves performing source coding to first compress the text and then channel coding to add robustness for the transmission across the channel. This approach is optimal in terms of minimizing end-to-end distortion with arbitrarily large block lengths of both the source and channel codes when transmission is over discrete memoryless channels. However, the optimality of this approach is no longer ensured for documents of finite length and limitations on the length of the encoding. We will show in this scenario that we can achieve lower word error rates by developing a deep learning based encoder and decoder. While the approach of separate source and channel coding would minimize bit error rates, our approach preserves semantic information of sentences by first embedding sentences in a semantic space where sentences closer in meaning are located closer together, and then performing joint source and channel coding on these embeddings.
1. INTRODUCTION
Separate source and channel coding is optimal with unlimited block lengths, but finite-length constraints motivate joint neural coding for text. The proposed architecture preserves sentence meaning and lowers word error rates at low bit budgets.
- 1. INTRODUCTION: Separation coding is optimal for memoryless and ergodic channels with infinite block lengths, but practical complexity and delay constrain this assumption.Source coding compresses data, while channel coding adds redundancy for error detection or correction.
- 1. INTRODUCTION: The work targets sentence transmission over erasure channels while recovering semantic information rather than requiring exact wording.Semantically equivalent outputs, such as “the car stopped” and “the automobile stopped,” count as error-free.
- 1. INTRODUCTION: The proposed neural system jointly encodes and decodes text using an RNN encoder, binarization layer, channel layer, and RNN decoder.The decoder may produce a different sentence that preserves the original semantic information content.
- 1. INTRODUCTION: At low bit budgets, the deep learning encoder-decoder outperforms separate source and channel coding in word error rate.The comparison uses Reed-Solomon channel coding with universal, Huffman, and 5-bit character source-coding baselines.
2. PROBLEM DESCRIPTION
The problem is to jointly design source and channel coding that preserves sentence meaning despite differing words and lengths in the recovered sentence. In practice, joint optimization is difficult, so source and channel coding are commonly treated separately.
- 2. PROBLEM DESCRIPTION: The system transmits a sentence as a fixed-length binary vector and receives a possibly nonbinary observation vector before sentence recovery.The sentence is represented as s, encoded by ϕℓ into b, and reconstructed as ŝ from receiver observations o.
- 2. PROBLEM DESCRIPTION: Traditional designs minimize word error rate and transmission bits, while treating source and channel coding separately because joint optimization is difficult.The passage frames separate design as a practical response to the difficulty of joint optimization.
- 2. PROBLEM DESCRIPTION: The target is a joint coding scheme that preserves meaning even when transmitted and recovered sentences use different words and lengths.This definition makes semantic fidelity the central objective rather than exact sentence matching.
3. DEEP LEARNING ALGORITHM
The proposed system jointly encodes and decodes sentences through recurrent networks, binary representations, and a differentiable erasure-channel layer. It trains the complete transmission pipeline end to end and decodes estimated sentences from channel observations.
- System architecture: The encoder accepts a sentence ending with <eos> and produces an ℓ-bit vector, while the channel generates observations that the decoder maps to an estimated sentence.The channel effects are random, and the decoder receives the channel output as input.
- Encoder: Word embeddings initialized with GloVe are processed by stacked bidirectional LSTM networks to obtain sequence representations and cell states.The encoder uses the embeddings of the sentence words and the special end-of-sentence symbol as BLSTM inputs.
- Encoder: The encoder concatenates the final outputs and cell states from all BLSTM stacks before converting each representation into a binary vector of length ℓ/2.The concatenation uses the final time step from every stack; the two resulting vectors together form the binary representation.
- Binarization: A stochastic binarization function maps values from [−1, 1] to {−1, 1} during training, with gradients passed through unchanged using the expectation.After training, the stochastic function is replaced by the deterministic function 2u(x) −1.
- Channel: The erasure channel is implemented as a dropout layer in which each bit is independently dropped with probability pd, producing observations in {−1, 0, 1}.The value 0 indicates an erased bit, and the channel is intended to model dropped packets, deep fades, or burst errors.
- Decoder: The decoder uses stacked LSTMs initialized from the received observation vector and feeds previously estimated word embeddings into subsequent decoding steps.Training initially uses the correct previous word, gradually annealing toward estimated words; deployment uses estimated words and beam search.
4. RESULTS
The experiments compare deep-learning joint source-channel coding with separate coding baselines on bit-erasure channels. The joint approach is most competitive under constrained bit budgets, channel stress, and longer sentences, while encoding sentence meaning through semantic embeddings.
- The experiments compare the deep-learning approach with traditional information-theoretic baselines for bit-erasure channels.Separate coding uses Reed-Solomon channel coding with universal compression, Huffman coding, or fixed-length character encoding.
- At a bit-erasure probability of 0.05, deep learning is most competitive with limited bit allocations, while baseline word error rates fall as redundancy increases.Among traditional baselines, gzip outperforms Huffman coding, which outperforms fixed-length encoding; baselines produce no errors when allocation exceeds required bits.
- With 400 bits per sentence, deep learning outperforms the baselines in stressed environments with low bit allocations and large bit-erasure rates.At this allocation, sentences cannot be completely encoded even with a lossless channel.
- Varying sentence lengths expose a limitation of fixed-size deep-learning embeddings, which can be improved by adapting embedding length to sentence length.Traditional baselines can allocate more bits to long sentences, whereas the deep-learning approach uses the same allocation regardless of sentence length.
- For sentences longer than 20 words, deep learning significantly outperforms baselines under 400 bits per sentence and a 0.05 erasure rate.Short sentences lie in the excess-bit-allocation regime, while some apparent word errors reflect synonym substitutions or meaning-preserving rephrasing.
- Deep learning produces semantic embeddings whose Hamming distances place sentences expressing the same idea closer together than information-theoretic baselines do.The embeddings are visualized with a dissimilarity matrix and multidimensional scaling.
5. CONCLUSION
The paper studies deep-learning joint source-channel coding for text when exact sentence recovery is unnecessary. It reports improved performance over separate coding with few bits per sentence, while identifying fixed bit length across sentence lengths as a drawback.
- The work addresses joint source-channel coding of text when conveying the sentence’s main information matters more than recovering its exact wording.
- The proposed joint source-channel coding scheme outperforms separate source and channel coding, especially with a small number of bits per sentence.
- The current algorithm uses a fixed bit length for sentences of different lengths, which the authors identify as a drawback and future-work target.