Source-linked AI summary
Efficient Audio-Visual Speech Enhancement via Reinforcement Learning-Based Neural Architecture Search
Khubaib Ahmed, Ahsan Adeel, Nasir Saleem, Kia Dashtipour, Ahsan Ulhaq, Amir Hussain
TL;DR
Audio-visual speech enhancement can improve noisy speech but is often too computationally expensive for edge devices, and multimodal architecture search must preserve stream alignment. The paper uses constrained RL-based architecture search to balance enhancement and model size, finding a compact model with 14.5 dB SI-SNR improvement and nine-fold compression.
Problem
State-of-the-art speech enhancement models can exceed 22 million parameters, while multimodal NAS must handle audio-visual dependencies, temporal alignment, and costly candidate evaluation.
Source: §1 · p. 1Method
The paper applies RL-based NAS with a constrained, temporally aligned search space and a reward that balances enhancement performance against model size.
Source: §1 · p. 1, §3.1 · p. 3Results
Takeaways & Limitations
The findings support RL-guided search as a blueprint for balancing performance and deployment constraints in multimodal systems.
Source: §5 · p. 7
Abstract
from arXiv · showhide
Audio-visual speech enhancement (AVSE) improves speech intelligibility in noisy environments but often relies on computationally expensive models unsuitable for resource-constrained edge devices. This paper introduces a reinforcement learning-driven neural architecture search (RL-NAS) framework to discover compact, efficient AVSE architectures. The search space preserves temporal alignment between audio and visual streams, while a novel reward function jointly optimises speech enhancement performance, measured by SI-SNR improvement, and model compactness. Using proximal policy optimisation with constraint-aware action masking, the framework evaluates only 35 candidate architectures and identifies a model with 2.9 million parameters, compared with 22 million in the baseline. The proposed architecture achieves a 14.5 dB SI-SNR improvement, only 0.7 dB below the baseline, while reducing model size by approximately nine-fold. Evaluation on commercial mobile CPUs demonstrates a 4.3-fold reduction in real-time factor, supporting practical on-device deployment. Compared with random search, RL-NAS achieves 92% higher search efficiency. Architectural analysis reveals that shallower but wider temporal convolutional blocks and aggressive visual pathway pruning can maintain enhancement performance while substantially reducing computational requirements. These findings demonstrate the potential of reinforcement learning-guided architecture search for developing lightweight, efficient and deployable multimodal speech enhancement systems.
1. Introduction
Audio-visual speech enhancement can use visual lip movements to improve noisy-speech processing, but state-of-the-art systems often exceed 22 million parameters, creating deployment barriers. This paper proposes RL-based architecture search to balance enhancement quality and compactness while preserving temporal alignment.
1. Introduction
- Audio-visual speech enhancement uses complementary visual lip-movement information to augment noisy acoustic signals.The introduction describes AVSE as particularly useful under severe acoustic degradation.Source: §1 · p. 1
- State-of-the-art AVSE systems often exceed 22 million parameters, limiting deployment on smartphones, hearing aids, and other resource-constrained edge devices.The paper identifies computational burden as a barrier to practical adoption.Source: §1 · p. 1
- Multimodal NAS must manage a large search space, preserve audio-visual temporal synchronization, and account for the compounded cost of evaluating candidates.Audio and visual pathway interdependencies introduce architectural constraints.Source: §1 · p. 1
- The proposed RL-NAS framework combines an alignment-preserving constrained search space, a parameter-performance reward, and progressive shrinking to find compact AVSE architectures.The introduction reports a 2.9-million-parameter model with 95.4% of baseline performance and a nine-fold size reduction.Source: §1 · p. 1
2. Related Work
Prior work develops NAS methods for general architectures, speech and audio, and multimodal systems, while identifying design patterns and efficiency challenges. The paper positions its contributions against gaps in multimodal robustness, quantization-aware search, and hardware-aware speech NAS.
2.1. Neural Architecture Search Foundations
- NAS evolved from RL and evolutionary search toward differentiable, parameter-sharing, hardware-aware, predictor-based, training-free, and multi-objective approaches.Early methods could require thousands of GPU-days, motivating more efficient search.Source: §2.1 · p. 2
2.2. NAS Applications in Speech and Audio Processing
- NAS applications in speech include ASR and time-delay networks, while temporal convolutional networks offer long-range modeling with computational efficiency for speech enhancement.NAS-TDNN is reported to achieve 96% model compression while preserving accuracy.Source: §2.2 · p. 2
2.3. Optimal Temporal Convolutional Network Design
- Prior TCN studies favor shallower, higher-expansion networks, linear dilation schedules, and shifting parameters toward earlier layers for noise robustness.The cited depth-width tradeoff typically uses expansion ratios of 1.5 to 2.0.Source: §2.3 · p. 2
2.4. Multimodal and Efficient Neural Architecture Search
- Multimodal NAS must coordinate fusion and modality investments; visual pathways tolerate 50%–80% complexity reductions, and additive fusion can reduce parameters by approximately 85% versus concatenation.Architecture embedding vectors have also been proposed for knowledge transfer across vision and audio tasks.Source: §2.4 · p. 2
- Pruning and quantization have demonstrated parameter reductions, while mobile architectures and newer search methods target further compression in low-parameter regimes.The cited emerging directions include binarized architecture search and normalizing flow-based design.Source: §2.4 · pp. 2–3
2.5. Generative Approaches to Speech Enhancement
- GAN-based AVSE uses adversarial training, but GANs add complexity and training instability compared with supervised approaches; this framework instead uses discriminative learning with efficiency constraints.Source: §2.5 · p. 3
2.6. Research Gaps and Our Contribution
- The paper identifies gaps in noise and SNR generalization, quantization-aware speech NAS, and hardware-aware search, then proposes alignment constraints, balanced rewards, and hardware-aware design.Its analysis also aims to provide architectural patterns for future multimodal speech-enhancement research.Source: §2.6 · p. 3
3. RL-Based NAS Methodology
The methodology casts audio-visual speech enhancement architecture search as constrained reinforcement learning, preserving temporal synchronization while balancing enhancement quality, model size, and architectural validity.
3.1. Constrained Search Space Design
- The search space preserves audio-visual temporal alignment and limits channel and compute dimensions; it reduces possibilities from O(1014) to 324 valid configurations.The stated reduction in search complexity is 99.9997%.Source: §3.1 · p. 3
3.2. Reinforcement Learning Formulation
- The RL state tracks architecture and performance, while constraint-aware action masking restricts choices to valid transitions and is reported to yield about 40% faster convergence than penalty-only approaches.State dimensions are normalized to [0,1] and include performance-aware components.Source: §3.2 · p. 3
3.2.1. Multi-Objective Reward Function Design
- The reward balances enhancement fidelity, parameter efficiency, and constraint enforcement, penalizing model size only above the 2.9-million-parameter target.Fidelity is normalized by baseline performance; constraint violations receive a severe penalty.Source: §3.2.1 · pp. 3–4
3.3. Policy Network Architecture
- A two-layer LSTM with 256 hidden units uses separate actor and critic heads, and learned embeddings of the 22-million-parameter baseline seed the search.The actor produces action probabilities; the critic estimates state value for advantage calculation.Source: §3.3 · p. 4, Figure 1 · p. 5
3.4. Training with Proximal Policy Optimization
- PPO trains the agent with clipped policy updates; the clipping region is ϵ = 0.2, and four parallel workers evaluate architectures.Training also uses ϵ-greedy exploration with ϵ = 0.3 and constraint-aware masking.Source: §3.4 · p. 4
3.5. Progressive Shrinking Strategy
- Progressive shrinking starts from the 22-million-parameter baseline, changes one architectural dimension per step, and estimates full-training SI-SNR from intermediate 10-epoch results.The estimate uses linear regression fit to historical data to reduce evaluation cost.Source: §3.5 · p. 4
3.6. Computational Complexity Analysis
- 35 architecture evaluations achieve 92% search efficiency compared with random exploration; total search cost is 1,750 epoch-equivalents, about 3× faster than generic NAS methods.The constrained design space and progressive strategy make architecture search feasible for multimodal speech enhancement.Source: §3.5 · p. 4, §3.6 · p. 4
4. Experiments and Results
On CEC2, the 2.9-million-parameter model achieves 14.5 dB SI-SNR improvement—95.4% of the 22-million-parameter baseline—with 87% fewer parameters. The experiments trace the architecture’s progressive shrinking and report comparisons with alternative models and mobile inference timing.
4.1. Experimental Setup
- Experiments use synchronized LRS3 audio-video, CEC2 noises at −5 to 5 dB SNR, SI-SNR improvement, and Intel i7-1185G7 mobile CPUs.The baseline is a 22-million-parameter audio-visual enhancement network.Source: §4.1 · p. 5
4.2. Architecture Evolution During Search
- After five optimization steps, reducing encoder channels from 256 to 192 cuts parameters by 26.5% while maintaining nearly identical SI-SNR performance.Later search modifications include bottleneck and TCN reductions, hidden-dimension scaling, and visual-pathway pruning.Source: §4.2 · p. 5, Table 2 · p. 6
4.3. Discovered Optimal Architecture
- The 2.9-million-parameter model uses three repeats of six-block TCNs and halves visual output channels from 256 to 128.Its TCN expansion ratio is 2.0; the bottleneck also decreases from 256 to 192 channels.Source: §4.3 · p. 5
4.4. Performance Comparison
- 14.5 dB SI-SNR improvement is 95.4% of the 15.2 dB baseline, with model size reduced by 87%.Table 3 benchmarks performance on the CEC2 test set.Source: §4.4 · pp. 5–6, Table 3 · p. 6
- At equal parameter budgets, RL search gains 0.6 dB over random search; mobile RTF is 0.19, processing 3-second files in about 570 milliseconds.The paper also reports 0.3 dB higher performance than a 6.1M-parameter handcrafted small model while using 2.1× fewer parameters.Source: §4.4 · p. 6
5. Conclusion
The paper concludes that constrained RL-based architecture search can produce a compact audio-visual speech enhancement model with performance close to a much larger baseline. Its architectural findings and proposed extensions concern multimodal efficiency and deployment.
Conclusion
- The 2.9-million-parameter model achieves 14.5 dB SI-SNR improvement, 95.4% of baseline performance, with nine-fold compression.The framework formulates search as a constrained MDP with state representations, action spaces, and multi-objective rewards.Source: §5 · p. 6
- The search identifies 128 visual channels and three repeats of six-block TCNs with expansion ratio 2.0 as compact architectural choices.The paper also reports that additive fusion outperformed concatenation-based alternatives while reducing parameters.Source: §5 · p. 6
- The authors identify real-time hearing aids, mobile video conferencing, and always-on voice assistants as potential deployment scenarios for the 11.7 MB model.The conclusion describes a four-fold reduction in computational requirements.Source: §5 · p. 6
- Future work includes dynamic architectures, quantization-aware NAS, cross-modal attention, self-supervised learning, generative NAS, and hardware-aware optimization.The authors also propose extending the framework to broader multimodal applications.Source: §5 · pp. 6–7