Source-linked AI summary
HI-TransPA: Hearing Impairments Translation Personal Assistant
Zhiming Ma, Shiyu Gan, Junhao Zhao, Xianming Li, Qingyun Pan, Peidong Wang, Mingjun Pan, Yuhao Mo, Jiajie Cheng, Chengxin Chen, Zhonglun Cao, Chonghan Liu, Shi Cheng
TL;DR
Hearing-impaired people can face communication barriers, and standard speech recognition struggles with atypical or partially articulated speech. HI-TransPA combines audio and lip dynamics in an assistant for translation and dialogue, using curated data and quality-aware curriculum learning. On HI-Dialogue, it is reported to outperform existing ASR methods, LALMs, and general Omni-Models.
Problem
Method
HI-TransPA is an instruction-driven audio-visual assistant that combines raw audio and lip dynamics for translation and dialogue, with data curation and quality-aware curriculum learning.
Source: §1 · p. 2Results
HI-TransPA consistently outperforms existing ASR methods, LALMs, and general Omni-Models on HI-Dialogue; its CER is 27%, the best among all baselines.
Source: §4.4 · p. 8, §5 · p. 9Takeaways & Limitations
The results highlight the potential of instruction-tuned multimodal intelligence for inclusive communication.
Source: §5 · p. 9The evaluation omits conversational fluency and empathy metrics, focusing instead on upstream comprehension of noisy audio-visual signals.
Source: §4.2 · p. 6
Abstract
from arXiv · showhide
Hearing-impaired individuals often face significant barriers in daily communication due to the inherent challenges of producing clear speech. To address this, we introduce the Omni-Model paradigm into assistive technology and present HI-TransPA, an instruction-driven audio-visual personal assistant. The model fuses indistinct speech with lip dynamics, enabling both translation and dialogue within a single multimodal framework. To address the distinctive pronunciation patterns of hearing-impaired speech and the limited adaptability of existing models, we develop a multimodal preprocessing and curation pipeline that detects facial landmarks, stabilizes the lip region, and quantitatively evaluates sample quality. These quality scores guide a curriculum learning strategy that first trains on clean, high-confidence samples and progressively incorporates harder cases to strengthen model robustness. Architecturally, we employs a novel unified 3D-Resampler to efficiently encode the lip dynamics, which is critical for accurate interpretation. Experiments on purpose-built HI-Dialogue dataset show that HI-TransPA achieves state-of-the-art performance in both literal accuracy and semantic fidelity. Our work establishes a foundation for applying Omni-Models to assistive communication technology, providing an end-to-end modeling framework and essential processing tools for future research.
1. Introduction
Hearing loss can impede verbal communication and participation, while standard assistive systems offer little support for hearing-impaired people expressing themselves. HI-TransPA brings translation and dialogue together in an audio-visual assistant, supported by multimodal data curation and quality-aware curriculum learning.
1. Introduction
- Audio-visual multimodal models have demonstrated capabilities in perception, reasoning, and generation, motivating their use for real-world communication challenges.The passage situates the work amid advances in machine translation, human–computer interaction, and content creation.Source: §1 · p. 1
- More than 1.5 billion people worldwide live with some degree of hearing loss, and over 430 million require rehabilitative support for moderate or higher impairment.The passage links hearing loss to difficulties in verbal communication, educational and occupational participation, information access, and social interaction.Source: §1 · p. 2
- Standard speech recognizers [3] [9] struggle with atypical or partially articulated utterances, while existing assistive technologies offer little support for users expressing themselves.The passage identifies a need for adaptive multimodal models that can process diverse, non-standard speech.Source: §1 · p. 2
- HI-TransPA combines translation and chat in one instruction-driven audio-visual assistant designed for hearing-impaired users.It processes raw audio and high-frame-rate lip dynamics and is described as understanding context and inferring user intent.Source: §1 · p. 2
- Its data pipeline extracts and aligns lip regions, uses multimodal metrics to split samples into easy and hard subsets, and supports training from easy to hard.The authors describe the quality-aware curriculum as intended to strengthen generalization across diverse hearing-impaired speech patterns.Source: §1 · p. 2
2. Related Work
Related work traces a shift from speech recognition toward audio-language and unified multimodal models, while noting limited assistive applications and scarce aligned speech-and-lip data. Hearing-support research has largely emphasized perception; HI-TransPA instead aims to support both translation and interactive dialogue.
2.1. ASR and Multimodal Large Language Models
- Audio research evolved from ASR toward LALMs supporting instruction following and spoken dialogue, then Omni-Models unifying text, audio, and vision.The passage notes that assistive applications of Omni-Models remain limited and that multimodal datasets pairing indistinct speech with lip dynamics are scarce.Source: §2.1 · pp. 2–3
2.2. AI for Hearing-Impaired Assistance
- Hearing-support research includes sign recognition and translation, lip reading, and lip-to-speech synthesis, but speech-expression tools have received comparatively little attention.Prior examples include multimodal sign processing, textual-visual sign embedding alignment, lip-reading knowledge distillation, and speech reconstruction from silent lip videos.Source: §2.2 · p. 3
- HI-TransPA uses the Omni-Model paradigm to pursue an audio-visual assistant for both accurate translation and interactive dialogue, supporting bidirectional communication.The passage contrasts this aim with approaches that treat people with hearing loss mainly as recipients of translation output.Source: §2.2 · pp. 3–4
3. Method
HI-TransPA addresses noisy, heterogeneous hearing-impaired speech and limitations in modeling high-frame-rate lip dynamics through data curation, a specialized vision architecture, and staged training. The method combines audio-visual processing with instruction fine-tuning for translation and dialogue.
3.1. Data Preprocessing and Curation
- The preprocessing pipeline extracts lip landmarks from facial keypoints, then aligns and stabilizes crops to normalize head motion and emphasize speech-relevant lip dynamics.It begins with 468 detected facial keypoints and retains those associated with the lip region.Source: §3.1 · p. 4
- Audio-visual samples are scored using audio transcription consistency and SNR alongside video motion magnitude, then divided into accepted and rejected subsets.Rejected samples are later reused as hard examples in curriculum learning.Source: §3.1 · pp. 4–5
3.2. Model Architecture
- The vision path encodes lip video as patch tokens, then uses a 64-query 3D-Resampler to reduce sequence length while preserving spatiotemporal cues.The architecture builds on Qwen2.5-Omni-3B and integrates SigLIP with the Unified 3D-Resampler.Source: §3.2 · p. 5
3.3. Multi-Stage Alignment and Fine-Tuning
- Training aligns visual features, co-adapts audio and vision for AVSR, then instruction-fine-tunes translation and dialogue modes end to end.The final stage uses paired audio-visual samples for /translate and generated conversational replies for /chat.Source: §3.3 · p. 5
3.4. Training Objective
- The easy-to-hard schedule is reported to stabilize convergence and yield consistent performance across diverse acoustic and visual quality levels.Training proceeds from accepted high-quality samples to the harder rejected subset.Source: §3.4 · pp. 5–6
3.5. Metrics Definition
- CER measures literal transcription accuracy, while EmbSim uses cosine similarity to assess semantic alignment between predicted and reference embeddings.Lower CER and higher EmbSim indicate better literal and semantic fidelity, respectively.Source: §3.5 · p. 6
4. Experiments
Experiments on HI-Dialogue evaluate literal and semantic comprehension across audio-only and multimodal systems, using a score that combines transcription accuracy and semantic similarity. Curriculum-trained HI-TransPA leads the reported comparisons, while ablations and qualitative examples examine the roles of lip input and comprehension in generating dialogue.
4.1. Experimental Setup
- The evaluation uses 9,673 HI-Dialogue samples from six volunteers, with 7,736 training and 1,937 test samples; training is split into accepted and rejected subsets for curriculum learning.The accepted subset has 4,733 samples and the rejected subset 3,003; training responses are distilled for instruction tuning.Source: §4.1 · p. 6
4.2. Evaluation Metrics
- CS balances literal accuracy, measured by 1 − CER, and semantic consistency, measured by EmbSim, but excludes conversational fluency and empathy.EmbSim is cosine similarity between predicted and reference embeddings; the metric focuses on upstream comprehension for comparability across models.Source: §4.2 · p. 6
4.3. Baselines
- The benchmark spans audio-only ASR, unimodal audio-language models, and multimodal Omni-Models, with baselines fine-tuned on HI-Dialogue for comparability.The authors frame audio-only systems as acoustic references and generic Omni-Models as multimodal baselines that can use visual cues.Source: §4.3 · pp. 6–7
4.4. Main Results and Analysis
- CS 0.79, EmbSim 0.84, and CER 27% make curriculum-trained HI-TransPA best among baselines, improving CER from 42% and EmbSim from 0.75.The comparison is on the HI-Dialogue test set.Source: Table 1 · p. 7, §4.4 · pp. 7–8, §4.5 · p. 8
4.5. Interpretation of the Comprehensive Score
- CS places models in (1 − CER) versus EmbSim space, where upper-right positions indicate lower error and stronger semantic consistency.The score is intended to reflect both text-level correctness and meaning preservation.Source: Figure 5 · p. 7, §4.5 · p. 8
4.4. Main Results and Analysis
- HI-TransPA’s 3B model reaches CS 0.70, exceeding the 7B Qwen2.5-Omni; adding curriculum learning raises CS to 0.79, EmbSim to 0.84, and lowers CER to 27%.The authors attribute the gain to specialized high-frame-rate lip modeling and curriculum learning.Source: §4.4 · pp. 7–8
4.6. Ablation Studies
- Removing visual input lowers CS from 0.70 to 0.64 and raises CER from 37% to 46%; curriculum training raises CS from 0.70 to 0.79 and lowers CER from 37% to 27%.The authors interpret these ablations as evidence that lip motion provides useful comprehension cues and progressive training improves robustness to noisy, difficult samples.Source: §4.6 · p. 8
4.7. Qualitative Analysis
- HI-TransPA’s near-perfect translation supports a helpful response about an international grain shipment, while a baseline’s misinterpretation yields nonsensical chat.The contrast illustrates how dialogue quality depends on the underlying comprehension.Source: §4.7 · pp. 8–9
5. Conclusion
HI-TransPA combines speech and high-frame-rate lip cues in an audio-visual Omni-Model for people with hearing loss. On HI-Dialogue, it outperforms existing ASR methods, LALMs, and general Omni-Models.
5. Conclusion
- HI-TransPA consistently outperforms existing ASR methods, LALMs, and general Omni-Models on the HI-Dialogue dataset.The conclusion describes its performance on multimodal real-world scenarios.Source: §5 · p. 9
- The model jointly uses speech and high-frame-rate lip cues with a redesigned visual encoder and quality-aware curriculum strategy.It is presented as an instruction-driven audio-visual Omni-Model for natural dialogue and precise speech translation.Source: §5 · p. 9
- The results highlight the potential of instruction-tuned multimodal intelligence for inclusive communication.Source: §5 · p. 9