Source-linked AI summary
Enhancing Neural Speech Coding with Semantic and Visual Cues
Yao Guo, Yang Ai, Hui-Peng Du, Xiao-Hang Jiang, Chen-Yuan Ning, Zhen-Hua Ling
TL;DR
At low bitrates, speech-only neural codecs may not preserve all contextual and articulatory information needed for high-quality reconstruction. SVSC adds semantic and visual branches, fuses their features through cross-attention, and uses fusion or distillation according to cue availability. The fusion mode raises ViSQOL from 3.86 to 4.01, while distillation raises it from 3.86 to 3.95 and supports speech-only inference.
Problem
At low bitrates, speech-only neural codecs have limited capacity to preserve phonetic, speaker, prosodic, acoustic, contextual, and articulatory information.
Method
SVSC adds semantic and image analysis-synthesis branches, fuses their features through cross-attention, and uses concatenation or distillation for different inference scenarios.
Results
ViSQOL improves from 3.86 to 4.01 in fusion mode and from 3.86 to 3.95 in distillation mode versus MDCTCodec.
Takeaways & Limitations
Semantic and visual cues provide complementary information, while distillation enables a speech-only codec when auxiliary cues are unavailable or inconvenient.
Abstract
from arXiv · showhide
At low bitrates, neural speech codecs have limited capacity to encode all information needed for high-quality re construction, especially when relying solely on speech-derived representations. To address this limitation, this paper proposes a Semantic- and Visual-enhanced Speech Codec (SVSC), which in corporates semantic and visual cues into the neural speech coding process. Specifically, built upon a mainstream neural speech cod ing architecture, SVSC introduces a semantic encoding-decoding branch and an image analysis-synthesis branch. It fuses deep semantic features with visual cues through a cross-attention mech anism, forming an auxiliary high-level representation enriched with contextual and articulatory information. To handle different inference scenarios, SVSC introduces two information-injection strategies based on the availability of auxiliary semantic and vi sual cues. When such cues are available, the fusion mode directly incorporates the auxiliary representations into the speech coding branch through feature concatenation; otherwise, the distillation mode transfers auxiliary information into the speech coding branch through knowledge distillation during training, enabling speech-only inference without additional inputs. Experimental results validate the effectiveness of incorporating semantic and visual cues, improving the ViSQOL score of reconstructed speech from 3.86 to 4.01.
I. INTRODUCTION
Low-bitrate neural codecs must preserve diverse speech information in compact codes, but speech-only representations may lose contextual and articulatory cues. SVSC addresses this bottleneck by combining semantic and visual information with two inference strategies.
- Low-bitrate quantized codes must preserve phonetic content, speaker characteristics, prosody, and acoustic detail in compact representations.
- Speech-only representations may insufficiently preserve high-level contextual and articulatory information, limiting reconstructed speech quality and naturalness.
- Semantic cues from pretrained speech models provide contextual and semantic information beyond local acoustic frames.
- Visual cues from lip and articulatory movements are correlated with speech content and speaker characteristics.
- SVSC fuses semantic and visual cues through cross-attention and supports direct concatenation or knowledge distillation for different inference scenarios.
II. PROPOSED METHOD
SVSC extends an MDCTCodec-based speech coding branch with semantic and image-processing branches. It injects fused auxiliary features directly when cues are available or transfers them during training for speech-only inference.
- II. PROPOSED METHOD: SVSC extends the MDCTCodec framework with semantic encoding-decoding and image analysis-synthesis branches.
- II. PROPOSED METHOD: The semantic branch extracts contextual cues, while the image branch extracts articulatory cues from lip image sequences.
- II. PROPOSED METHOD: In fusion mode, auxiliary representations are directly injected into the speech coding branch through feature concatenation when cues are available.
- II. PROPOSED METHOD: In distillation mode, auxiliary information is transferred into the speech encoder during training, enabling speech-only inference.
- II. PROPOSED METHOD: The speech branch processes MDCT spectra, quantizes encoded multimodal features with RVQ, and reconstructs waveforms through inverse MDCT.
C. Auxiliary-Modality Feature Processing Branches
The auxiliary branches transform speech-model outputs and lip-image sequences into aligned semantic and visual features while reconstructing their inputs during training. These branches supply modality-specific information for multimodal speech coding.
- 1) Semantic Feature Encoding-Decoding Branch:: The pretrained speech model receives speech downsampled from 48 kHz to 16 kHz and produces an output sequence P.
- 1) Semantic Feature Encoding-Decoding Branch:: The semantic encoder maps P to a compact semantic feature, and the semantic decoder reconstructs the pretrained model’s output space.
- 1) Semantic Feature Encoding-Decoding Branch:: The semantic feature is temporally expanded by frame replication to match the MDCT spectrum’s frame count.
- 1) Semantic Feature Encoding-Decoding Branch:: Semantic reconstruction uses mean squared error between the reconstructed representation and the original pretrained-model output.
- Auxiliary-Modality Feature Processing Branches: The image branch processes time-aligned lip-image sequences into visual features and reconstructs the image sequence during training.
2) Image Analysis-Synthesis Branch:
The image analysis-synthesis branch extracts visual features from lip-image sequences and reconstructs those sequences during training. Its reconstruction objective is defined as image mean squared error.
- 2) Image Analysis-Synthesis Branch:: The image analyzer uses cascaded analysis blocks to extract features from lip-image sequences, while the synthesizer restores their spatial resolution.
- 2) Image Analysis-Synthesis Branch:: During training, image reconstruction loss is defined as the mean squared error between the input image sequence and its reconstruction.
D. Multimodal Feature Fusion
The feature fusion module combines semantic and visual features through cross-attention, then injects the resulting bimodal representation into the speech coding branch either directly or through distillation. This supports higher-quality multimodal inference and speech-only inference when auxiliary cues are unavailable.
- Multimodal Feature Fusion: Cross-attention uses semantic features as keys and values and visual features as queries to produce a bimodal high-level representation.The resulting feature is combined with an intermediate spectral feature from the speech encoder.
- Fusion Mode: The fusion mode concatenates the speech feature X3 with the bimodal feature and projects the result back to X3's dimension.The concatenated feature is fed into the subsequent speech-encoder block.
- Distillation Mode: The distillation mode uses the multimodal feature as a teacher-side representation and trains the speech feature X3 to approach it.This transfers auxiliary information into the speech encoder during training.
- Inference Availability: After distillation, the auxiliary branches and fusion module can be removed, enabling speech-only coding without additional inputs or computational overhead.This is the deployment advantage of the distillation mode when auxiliary cues are unavailable or inconvenient.
E. Training Criteria
SVSC retains the MDCTCodec loss while adding semantic and image reconstruction supervision, with an additional distillation loss for the distillation mode. The objectives therefore differ according to whether auxiliary information is directly fused or transferred during training.
- Training Criteria: SVSC training retains the original MDCTCodec loss and supervises the semantic and visual branches with semantic and image reconstruction losses.The losses are denoted LMDCTCodec, LS, and LI, respectively.
- Fusion Mode: The fusion-mode objective combines the MDCTCodec, image reconstruction, and semantic reconstruction losses.The formulation is introduced specifically for fusion-mode training.
- Distillation Mode: The distillation-mode objective further introduces the distillation loss LD to transfer auxiliary information into the speech encoder.The added loss distinguishes distillation-mode training from fusion-mode training.
- Loss Weighting: The hyperparameters λI, λS, and λD control the weights of the different loss terms.These weights govern the relative contributions of the image, semantic, and distillation losses.
A. Datasets
The experiments use the TaL corpus, focusing on the TaL80 subset of multi-speaker recordings containing speech, ultrasound tongue imaging, and optical lip videos. Only speech and lip-video data are used in the experiments.
- Dataset: The TaL80 subset contains recordings from 81 native English speakers without voice talent.Each utterance includes ultrasound tongue imaging, optical lip videos, and speech in the broader TaL corpus.
- Modalities: The experiments use only the speech and lip-video data from TaL80.The dataset also contains ultrasound tongue imaging, but it is not used in these experiments.
- Recording Settings: Lip videos are recorded at 60 fps, while speech is recorded at 48 kHz with 16-bit depth.These are the stated recording settings for the selected modalities.
B. Experimental Settings
The experimental setup uses a 6 kbps MDCTCodec-based speech branch at 48 kHz, WavLM semantic features, and specified convolutional and optimization settings. Evaluation compares SVSC with advanced neural speech codec baselines on the test set.
- Speech Coding Setup: The speech coding branch follows MDCTCodec at 6 kbps and 48 kHz, with MDCT features X3 of dimension DM = 256.The MDCT frame shift is 40, producing a 1.2 kHz speech-feature frame rate.
- Semantic Features: WavLM extracts semantic features after the 48-kHz waveform is downsampled to 16 kHz.The semantic feature dimension is set to Dp = 768.
- Semantic Branch: The semantic branch maps the WavLM feature dimension from 768 to 256 and then to Ds = 64.It uses an initial 1D convolution followed by encoder blocks and a linear projection.
- Evaluation: Table I evaluates SVSC and advanced neural speech codecs on the test set.The table caption defines the comparison scope but does not specify individual metric values.
- Optimization: Training uses batch size 16 on one NVIDIA RTX 4090D GPU with AdamW optimization.The initial learning rate is 2 × 10^-4 and decays by 0.999 after each epoch.
C. Baselines and Evaluation Metrics
SVSC is evaluated against representative neural speech codecs using perceptual, intelligibility, and distortion-related metrics. It improves over the MDCTCodec backbone in both inference modes, with fusion achieving the strongest reported quality and distillation enabling speech-only inference.
- Comparison with Advanced Codecs: PESQ rises from 3.25 to 3.37 and ViSQOL from 3.86 to 4.01 in fusion mode versus the MDCTCodec backbone.Fusion achieves the best or tied-best performance on most objective metrics when auxiliary cues are available during inference.
- Comparison with Advanced Codecs: Distillation improves ViSQOL from 3.86 to 3.95 over MDCTCodec while requiring only speech input during inference.It also improves PESQ, CSIG, CBAK, and COVL compared with the backbone.
- Comparison with Advanced Codecs: Fusion provides better reconstruction quality than distillation, while distillation offers a deployable speech-only codec when auxiliary cues are unavailable or inconvenient.The difference reflects direct use of auxiliary cues during inference in fusion versus implicit transfer during training in distillation.
- Comparison with Advanced Codecs: SVSC achieves competitive or superior performance across the evaluated objective metrics, alleviating the low-bitrate information bottleneck.The evaluation uses PESQ and ViSQOL for perceptual quality, STOI for intelligibility, and CSIG, CBAK, and COVL for distortion, background intrusiveness, and overall quality.
E. Ablation Studies
The ablation study evaluates the full SVSC against variants using only semantic or only visual cues. The full model performs best, showing that the two cue types provide complementary guidance for low-bitrate reconstruction.
- Ablation Studies: The ablation compares full SVSC with w/o Visual, which uses only semantic information, and w/o Semantic, which uses only visual information.The ablations are conducted in fusion mode to directly assess the contribution of each auxiliary cue.
- Ablation Studies: Full SVSC achieves the best performance across all objective metrics, indicating complementary semantic and visual information.Their joint use through cross-attention provides more effective guidance than either cue alone.
- Ablation Studies: Removing visual cues lowers ViSQOL from 4.01 to 3.98, while removing semantic cues lowers it to 3.85.The semantic-cue removal produces the larger degradation, with other metrics also declining.
- Ablation Studies: SVSC improves reconstruction quality over MDCTCodec, and joint semantic-visual use outperforms either cue alone on the TaL80 dataset.The conclusion also notes competitive performance against representative neural speech codecs.