Audiovisual joint learning for end-to-end hearing aids

I. INTRODUCTION

• Hearing-aid benefits remain limited in noise, reverberation, and competing speech, while separate enhancement and compensation stages may amplify residual interference alongside target speech.Audio-only processing may also struggle to separate target speech under strong interference or introduce distortions that degrade perceived quality.Source: §I · pp. 1–2
• AV-NeuroAMP outperformed competing baselines in objective and subjective noisy-condition evaluations, with promising cross-lingual generalization and perceptual gains particularly under competing speech.The evaluations included conventional compensation, audio-only NeuroAMP, and two-stage systems; AC-FiLM also outperformed direct audiogram concatenation.Source: §I · p. 2

A. NeuroAMP: audio-only neural amplification

• NeuroAMP maps noisy speech and a listener’s audiogram directly to amplified speech, trained against a target produced with NAL-R and WDRC.Its audiogram encoder maps a six-dimensional normalized audiogram to a representation concatenated with speech spectral features.Source: §II · p. 3, A. NeuroAMP: audio-only neural… · p. 3

B. AV-NeuroAMP: audiovisual neural amplification

• AV-NeuroAMP combines noisy-speech spectral features with visual features, then conditions the audiovisual representation on the listener’s audiogram.The framework adds a visual encoder and AC-FiLM to NeuroAMP and uses a multi-objective training loss.Source: B. AV-NeuroAMP: audiovisual neural… · p. 3, §II · p. 3

1. Visual feature extraction

• The visual encoder extracts face representations and passes them through an adapter to produce visual features aligned with the audio features.The facial extractor is pretrained and frozen during AV-NeuroAMP training, while the TCN-based adapter is jointly trained.Source: §1 · p. 3

2. Audiogram-conditioned feature-wise linear modulation (AC-FiLM)

• AC-FiLM combines frequency-dependent and frequency-shared audiogram conditions to generate scaling, shifting, and gating parameters for audiovisual features.Applying modulation after instance normalization and retaining a residual connection preserves audiovisual information while incorporating hearing-loss patterns and overall severity.Source: §2 · p. 5

3. Training target and loss function

• AV-NeuroAMP estimates a personalized enhancement and amplification target based on NAL-R and WDRC applied to clean target speech, using a combined multi-component loss.The loss combines consistency, time-domain, magnitude, complex-spectral, and phase losses for the dual-path magnitude and phase estimates.Source: §3 · pp. 5–6

1. Clarity audiograms

• Six thresholds at 250, 500, 1000, 2000, 4000, and 6000 Hz serve as model inputs and for the NAL-R prescription.The Clarity Challenge provides 272 ear-specific audiograms: 166 for training and 106 for testing.Source: §1 · p. 6

2. AVSEC-2: training and in-domain evaluation

• 34,524 audiovisual scene pairs form the AVSEC-2 training set, with 3,306 development scenes used for in-domain evaluation.Scenes include LRS3 speech, competing speakers, and environmental noise; audiogram conditioning yields 138,096 training pairs.Source: §2 · p. 6

3. TMSV: out-of-domain evaluation and subjective listening tests

• 6,000 noisy Mandarin samples from 240 TMSV utterances support cross-lingual objective evaluation and listening tests.The two talkers each contribute 120 phonetically balanced sentences; conditions include background talkers and four noise types at SNRs from 8 to −4 dB.Source: §3 · p. 6, C. Audiovisual datasets and audiograms · p. 6

D. Competing baseline systems

• The five systems compared are Conventional, audio-only and audiovisual two-stage systems, NeuroAMP, and AV-NeuroAMP.The two-stage systems apply NAL-R/WDRC amplification after SEMamba or AVMamba enhancement; NeuroAMP is the audio-only end-to-end baseline.Source: D. Competing baseline systems · pp. 6–7

E. Model implementation details

• Audio uses 16-kHz sampling and a 512-point STFT, while grayscale 224×224 video runs at 25 frames per second.Training randomly crops synchronized two-second segments of 32,000 audio samples and 50 video frames, zero-padding shorter samples.Source: E. Model implementation details · p. 7
• Eight time-frequency Mamba modules feed magnitude and phase decoders; training runs for 200 epochs with a global batch size of 16.The model uses AdamW and an exponentially decaying learning rate; inference processes complete speech without segmentation.Source: E. Model implementation details · p. 7
• 18.019 million parameters comprise 6.834 million trainable and 11.185 million frozen visual-front-end parameters.AC-FiLM adds 21,824 trainable parameters and under 0.01 GMACs, compared with 232.16 GMACs for a two-second audiovisual input.Source: E. Model implementation details · p. 7

F. Objective evaluation metrics

• HASPI v2 and HASQI v2 assess intelligibility and quality, respectively, using thresholds at six audiometric frequencies.The frequencies are 250, 500, 1000, 2000, 4000, and 6000 Hz.Source: F. Objective evaluation metrics · p. 7
• MSE measures waveform-level consistency with the personalized amplified target, while SNR evaluates overall noise reduction.Source: F. Objective evaluation metrics · p. 7
• MBGE measures short-time, band-specific signal-level consistency between the model output and target across six frequency bands.It uses a 512-point STFT, restores compressed magnitudes to the linear domain, and computes band levels at 250, 500, 1000, 2000, 4000, and 6000 Hz.Source: F. Objective evaluation metrics · p. 7
• Lower MBGE indicates greater consistency between the output and target in short-time, band-specific signal levels.For noisy inputs, the error can reflect both residual interference and deviation from prescribed amplification gain.Source: F. Objective evaluation metrics · pp. 7–8

G. Audiogram-conditioning ablation study

• The audiogram-conditioning ablation compares direct concatenation with AC-FiLM within the same audiovisual backbone, holding training and evaluation settings constant.Source: G. Audiogram-conditioning ablation study · p. 8

H. Listening tests

• Two listening experiments assess normal-hearing participants under simulated hearing loss and listeners with hearing loss.The section introduces the speech materials, processing conditions, and analysis used in both experiments.Source: H. Listening tests · p. 8

1. Test stimuli and processing

• 90 Mandarin test samples comprise clean speech, street-noise mixtures, and interfering-speech mixtures, with noisy samples at −1 and 5 dB SNR.Listeners heard processed audio only; visual information was not presented during testing.Source: §1 · p. 8

2. Listening procedure and statistical analysis

• Listeners rated quality from 0 to 5 and identified characters for intelligibility; each stimulus could be replayed once.Method and acoustic condition were analyzed with repeated-measures ANOVA, followed by condition-specific paired comparisons with Holm correction.Source: §2 · p. 8

3. Experiment 1: normal-hearing listeners under simulated hearing loss

• 29 normal-hearing participants completed a preliminary experiment using audiogram-based MSBG simulation after personalized amplification.The experiment was designed to verify the method against other methods under simulated hearing-loss conditions.Source: §3 · p. 8

4. Experiment 2: listeners with hearing loss

• The hearing-loss-user listening experiment tested 12 participants across 20 ears, using each tested ear’s audiogram for compensation.Eight participants were tested bilaterally and four unilaterally; the mean PTA4 across ears was 59.2 dB HL.Source: §4 · p. 9

1. In-domain performance

• Under clean input, AV-NeuroAMP’s HASPI and HASQI scores were 0.925 and 0.932, close to Conventional(SAmp) at 0.928 and 0.944.Relative to NeuroAMP, AV-NeuroAMP reduced MSE from 1.6 × 10−3 to 2.6 × 10−4 and MBGE from 1.30 dB to 0.79 dB.Source: §1 · p. 9
• Under noisy input, AV-NeuroAMP scored 0.780 on HASPI and 0.625 on HASQI, versus 0.419 and 0.268 for NeuroAMP.AV-NeuroAMP also reduced MSE from 2.6×10−2 to 1.1×10−3, reduced MBGE from 11.30 dB to 3.48 dB, and increased target reference SNR from −2.98 dB to 10.20 dB versus NeuroAMP.Source: §1 · p. 9

2. Band-specific gain error

• Under noisy input, AV-NeuroAMP reduced band-specific gain error by 6.21–9.16 dB versus NeuroAMP across frequency bands.NeuroAMP’s noisy-input errors ranged from 9.54 dB at 500 Hz to 13.12 dB at 6 kHz, while AV-NeuroAMP’s ranged from 3.4 dB at 1 kHz to 4 dB at 6 kHz.Source: §2 · p. 9, Figure 5 · p. 10

3. End-to-end versus two-stage processing

• On noisy in-domain input, AV-NeuroAMP obtained the best scores on all five metrics compared with the two-stage systems.For clean input, Two-stage(A) had the lowest MSE (7.8×10−5) and highest target-referenced SNR (23.81 dB) among learning-based systems, but lower HASPI and HASQI than both end-to-end systems.Source: §3 · pp. 9–10, Table II · p. 10

4. Out-of-domain performance

• On noisy Mandarin input, AV-NeuroAMP led all systems across the evaluation metrics, including HASPI 0.598 and HASQI 0.454.Its MSE was 3.4 × 10−3, MBGE was 6.04 dB, and target-referenced SNR was 7.35 dB.Source: §4 · p. 10

• Across both input conditions, AC-FiLM beat direct concatenation on all five metrics; noisy HASPI/HASQI rose from 0.738/0.563 to 0.780/0.625.For noisy input, MSE decreased from 1.4 × 10−3 to 1.1 × 10−3, MBGE from 4.15 dB to 3.48 dB, and target reference SNR increased from 8.85 dB to 10.20 dB.Source: §5 · pp. 10–11
• AC-FiLM reduced band-specific gain error by 0.16–0.38 dB under clean input and 0.48–0.83 dB under noisy input.The reported reductions span 500 Hz to 6 kHz for clean input and 250 Hz to 6 kHz for noisy input.Source: §5 · p. 11, Figure 6 · p. 11

1. Experiment 1: normal-hearing listeners under simulated hearing loss

• Under competing speech, AV-NeuroAMP significantly outperformed both Conventional and NeuroAMP in quality and intelligibility (p < 0.001).Its quality score was 2.23 ± 0.78 and intelligibility was 0.572 ± 0.195; NeuroAMP scored 1.34 ± 0.62 and 0.165 ± 0.188, respectively.Source: §1 · p. 11
• Under street noise, NeuroAMP and AV-NeuroAMP both significantly outperformed Conventional in speech quality and intelligibility, with no significant difference between the end-to-end systems.For AV-NeuroAMP, quality was 3.20 ± 1.04 and intelligibility was 0.836 ± 0.239.Source: §1 · p. 11

2. Experiment 2: listeners with hearing loss

• In clean speech, the three methods showed no significant differences in quality or intelligibility among hearing-aid users.The corresponding quality scores were 3.38, 3.26, and 3.28 for Conventional, NeuroAMP, and AV-NeuroAMP.Source: §2 · pp. 11–12
• Under street noise, AV-NeuroAMP exceeded NeuroAMP by 0.37 quality points and 0.116 normalized-intelligibility points, with all pairwise differences significant.Compared with Conventional, AV-NeuroAMP improved quality by 1.18 points and normalized intelligibility by 0.346.Source: §2 · p. 12
• Under competing speech, AV-NeuroAMP scored highest and NeuroAMP lowest for both quality and intelligibility; all pairwise differences were significant.AV-NeuroAMP improved quality and intelligibility over Conventional by 0.37 and 0.097 points, respectively, and over NeuroAMP by 0.68 and 0.237 points.Source: §2 · p. 12, Figure 8 · p. 13

IV. DISCUSSION

• The discussion concludes that AV-NeuroAMP's perceptual advantage depends on acoustic condition and is greatest under competing-speech interference.Across the reported listening tests, its benefits were most pronounced when speech competed with another speaker.Source: §IV · pp. 12–13

A. Perceptual benefits given by visual cues

• In clean speech, neither end-to-end method differed significantly from Conventional; under street noise, both improved quality and intelligibility, with AV-NeuroAMP providing further improvements.Under competing speech, AV-NeuroAMP ranked highest and NeuroAMP lowest; the authors attribute the audiovisual model's advantage to facial movements guiding target-speech extraction.Source: A. Perceptual benefits given by visual… · p. 13

B. End-to-end processing and out-of-domain performance

• AV-NeuroAMP outperformed evaluated audio-only and audiovisual two-stage systems across five in-domain objective metrics and had the best noisy-input performance across all five metrics on the out-of-domain Mandarin dataset.The Mandarin evaluation differed from training in language, speakers, and acoustic conditions.Source: B. End-to-end processing and… · pp. 13–14
• AV-NeuroAMP was not superior on every clean-input metric: the audio-only two-stage system had the lowest MSE and highest target-referenced SNR.The authors note that signal-level errors do not necessarily correspond to better predicted perceptual quality or more accurate short-time gain.Source: B. End-to-end processing and… · pp. 13–14

C. Audiogram conditioning and gain accuracy

• AC-FiLM outperformed direct audiogram concatenation on all five metrics for clean and noisy input, with lower gain errors across all evaluated frequency bands.Both systems used the same audiovisual backbone, training data, target, loss function, and evaluation protocol.Source: C. Audiogram conditioning and gain… · p. 14

D. Limitations and future work

• The framework assumes the supplied video corresponds to the intended speaker; multi-person conversations may require goal cues such as gaze, head orientation, or user choice.The authors propose incorporating such cues to adapt to communication goals as well as hearing loss.Source: D. Limitations and future work · p. 14
• NAL-R is linear, whereas current fittings typically use nonlinear prescriptions, so the results support learning selected processing goals rather than optimal clinical fitting.The authors call for evaluation of nonlinear prescriptions, personalized comfort constraints, alternative compression settings, and direct optimization based on perception or intelligibility.Source: D. Limitations and future work · p. 14
• The system processes complete utterances and is not designed for causal wearable operation; deployment requires controlled latency, memory use, and power consumption.The visual encoder accounts for about 41% of computational cost.Source: D. Limitations and future work · p. 14

V. CONCLUSIONS

• AV-NeuroAMP extends personalized neural amplification with target-talker video and audiogram-conditioned feature modulation to estimate a final listener-specific NAL-R/WDRC target.The Mamba-based end-to-end model jointly learns speech enhancement and hearing-loss compensation from noisy audio, synchronized video, and an audiogram.Source: §V · p. 14
• AV-NeuroAMP outperformed Conventional and audio-only NeuroAMP on all five objective metrics in both in-domain English and out-of-domain Mandarin evaluations.These results were obtained with noisy input.Source: §V · p. 14
• Under noisy in-domain conditions, AV-NeuroAMP outperformed two-stage SE–amplification cascades on all five objective metrics.The authors interpret this as support for directly estimating the final listener-specific target.Source: §V · p. 14
• AC-FiLM outperformed direct audiogram concatenation on all five objective metrics with in-domain clean and noisy input.It also achieved closer agreement with the listener-specific multiband gain target.Source: §V · p. 14
• In listeners with hearing loss, no significant method differences appeared for clean speech, while all pairwise quality and intelligibility comparisons were significant in street noise and competing speech.In street noise, AV-NeuroAMP scored highest, followed by NeuroAMP and Conventional; in competing speech, it scored highest, followed by Conventional and NeuroAMP.Source: §V · pp. 14–15
• Future work should develop causal, low-latency wearable implementations, conduct larger counterbalanced speech-recognition trials, and explore cochlear-implant and closed-loop neural or cognitive-conditioning applications.Source: §V · p. 15

DATA AVAILABILITY

• This part lists references, not conclusions or results.The entries cover related work and methods, but do not report this paper’s findings.Source: DATA AVAILABILITY · p. 17