PaperlayerView exact revision on arXiv

Source-linked AI summary

Proactive Hearing Assistants that Isolate Egocentric Conversations

Guilin Hu, Malek Itani, Tuochao Chen, Shyamnath Gollakota

arXiv:2511.11473v1cs.CLcs.SDeess.AS

TL;DR

Crowded, overlapping conversations can make it difficult to isolate relevant voices, while existing hearing assistants often require manual source selection. This paper uses self-speech and conversational dynamics to identify and separate the wearer’s partners; it reports partner-identification and speech-quality results in out-of-distribution and real-world settings.

  • Problem

    In crowded auditory environments, isolating relevant voices is difficult, and existing hearing assistants rely on manual source selection that is impractical in some multi-party conversations.

    Source: §1 · p. 1

  • Method

    The system anchors on wearer self-speech and uses turn-taking dynamics in a dual-model pipeline to identify and separate conversational partners from egocentric audio.

    Source: §1 · p. 2, §3.2 · p. 4, §3.2.1 · p. 4

  • Results

    Across out-of-distribution and real-world egocentric settings, the system reports 80–92% partner-identification accuracy, 1.5–2.2% confusion rates, and 7.22–11.95 dB (SISDRi) speech-quality improvement.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work demonstrates a path toward hearing assistants that infer whom the user wants to hear and adapt to conversation dynamics.

    Source: §6 · p. 9

    Cultural and linguistic differences in turn-taking may make language- or culture-specific fine-tuning useful for improving robustness.

    Source: §7 · p. 10

Abstract

from arXiv · show

We introduce proactive hearing assistants that automatically identify and separate the wearer's conversation partners, without requiring explicit prompts. Our system operates on egocentric binaural audio and uses the wearer's self-speech as an anchor, leveraging turn-taking behavior and dialogue dynamics to infer conversational partners and suppress others. To enable real-time, on-device operation, we propose a dual-model architecture: a lightweight streaming model runs every 12.5 ms for low-latency extraction of the conversation partners, while a slower model runs less frequently to capture longer-range conversational dynamics. Results on real-world 2- and 3-speaker conversation test sets, collected with binaural egocentric hardware from 11 participants totaling 6.8 hours, show generalization in identifying and isolating conversational partners in multi-conversation settings. Our work marks a step toward hearing assistants that adapt proactively to conversational dynamics and engagement. More information can be found on our website: https://proactivehearing.cs.washington.edu/

1 Introduction

The paper targets hearing assistance in crowded conversations, where manual source selection is impractical, by inferring the wearer’s conversation partners without prompts. It combines self-speech anchoring and turn-taking cues with dual-timescale processing, reporting partner-identification and speech-quality results in out-of-distribution and real-world settings.

1 Introduction

  • Existing hearing assistants require manual source selection, which becomes impractical when multi-party conversations involve spatially dispersed speakers or more than two speakers.The paper frames this as a challenge alongside real-time, on-device operation and generalization to real-world multi-party settings.Source: §1 · pp. 1–2
  • The system automatically identifies and enhances voices engaged with the wearer, suppressing others without explicit prompts.It processes egocentric binaural audio and adapts to conversational engagement.Source: §1 · p. 1
  • Wearer self-speech anchors partner tracking, while turn-taking cues such as alternating speech, low overlap, and temporal coordination guide selection.The assistant activates after the wearer speaks for a few seconds, signaling conversational intent.Source: §1 · p. 2
  • A fast model extracts the target conversation every 12.5 ms, while a slower model runs once per second to encode longer-term conversational dynamics.This dual-model design balances responsiveness with conversational context length.Source: §1 · p. 2
  • 80–92% partner-identification accuracy, 1.5–2.2% confusion rates, and 7.22–11.95 dB (SISDRi) speech-quality improvement are reported across out-of-distribution and real-world egocentric settings.The system also operates in real time on embedded and mobile devices.Source: §1 · p. 2

2 Related work

Prior work addresses conversation structure, speaker activity, auditory attention, and hearing augmentation, but the paper distinguishes its goal as real-time identification, separation, and enhancement of all speakers engaged with the wearer.

Related work

  • Unlike Target Conversation Extraction, this work targets real-time use without future context, anchors on egocentric self-speech rather than explicit speaker embeddings, and uses spatialized egocentric audio rather than monaural recordings.Target Conversation Extraction uses turn-taking cues to extract a target conversation.Source: §2 · p. 2
  • Prior attention and engagement approaches include audiovisual speaker detection, spatial localization, and egocentric-video tasks, but the cited video tasks typically stop at detection.This work instead identifies, separates, and enhances speakers engaged with the wearer in real time and on-device under real-world constraints.Source: §2 · p. 3
  • Auditory-attention decoding correlates brain activity with competing audio streams, while augmented-hearing systems use spatial filtering or manual enrollment; neither description matches the paper’s dialogue-based conversational tracking goal.The paper also distinguishes its task from proactive assistants focused on information-seeking or planning.Source: §2 · p. 3

3 Proactive Hearing Assistants

The system anchors partner tracking on the wearer’s self-speech and combines binaural egocentric audio with a dual-model pipeline for low-latency extraction and longer-range conversational context. Its training uses spatialized and synthetic mixtures, followed by staged training and on-device inference.

3.1 Problem formulation

  • The system must track speakers as they join or leave the target conversation, while rendering partner speech with low delay requires 10–20 ms chunks and on-device processing.The stated assumption is that the wearer actively participates; passive listening is a non-goal.Source: §3.1 · p. 3

3.2 Proactive assistant modeling

  • The fast streaming model processes 12.5 ms chunks, while the slow model uses mixture audio and self-speech to generate a conversation embedding every second that guides target-conversation extraction.The beamformer extracts the wearer’s voice from binaural audio; the fast model receives a single egocentric stream and the embedding.Source: Figure 2 · p. 4, §3.2 · p. 4

3.2.1 Dual-model processing

  • A high-latency attention-based model captures long sequences and supplies embeddings to a low-latency LSTM model, reducing the fast model’s memory footprint.The slow model’s processing is decoupled from streaming, but larger T values reduce responsiveness; both models use monaural input to reduce computation and focus on conversational dynamics.Source: §3.2.1 · p. 4

3.3 Training strategy

  • Training uses non-egocentric conversation datasets spatialized to emulate egocentric conditions, because existing egocentric datasets lack the third-person binaural recordings needed for mixture synthesis.Candor supplies 850 hours of English two-speaker conversations and RAMC supplies 180 hours of Mandarin two-speaker conversations.Source: §3.3 · pp. 4–5

3.3.1 Synthetic dataset creation

  • Five synthetic datasets cover two- and three-speaker conversations, speaker transitions into interfering conversation, and evaluation-only four- and five-speaker cases.They are created by aligning LibriTTS audio with RAMC conversation timestamps and assigning speakers to turns.Source: §3.3.1 · p. 5

3.3.2 Training procedure

  • Training proceeds through pretraining on synthetic and Candor mixtures, spatialized training with beamformer-generated self-speech, and finetuning that perturbs silence and overlap.The first stage jointly trains both models with a negative-SNR loss on the fast model’s target-conversation reconstruction.Source: §3.3.2 · p. 5

3.4 On-device real-time inference

  • On-device inference runs the fast model on an Orange Pi 5B and the slow model on Apple M2 silicon; average processing times are 8.9 ms per 12.5 ms chunk and 41.3 ms per one-second chunk.Peak streaming memory is 86.33 MB for the fast model and 591.47 MB for the slow model.Source: §3.4 · p. 5

4 Evaluation

Evaluation spans multiple metrics and test settings, with results on spatialized and real-world conversations, cross-setting generalization, subjective ratings, and ablations of conversational context and model design.

4.1 Metrics

  • SISDRi and ΔPESQ measure speech enhancement and quality gains; accuracy and confusion rate track correct-partner and interfering-speaker selection at each turn.Source: §4.1 · pp. 5–6

4.2 Testsets

  • Tests span Libri 2- and 3-speaker and speaker-leaving scenarios, Candor, RAMC, Japanese Duplex, and OOD SpokenWOZ, with no test speaker or turn overlap in training.Source: §4.2 · p. 6

4.3 Results

  • On spatialized synthetic Libri conversations emulating egocentric conditions, partner input SISDR drops from 1 to –10 dB and PESQ from 2.52 to 2.04; the model improves SISDR by 14.62 dB and PESQ by 0.63, with 96.8% selection accuracy and 0.8% confusion.Source: §4.3 · p. 7
  • Across open-source conversational tests, the dual-model consistently outperforms DeepFilterNet2 across four metrics, while the baseline’s uniform enhancement fails to deliver SISDR improvements.Source: §4.3 · p. 6, Table 1 · p. 7
  • Comparable results on unseen 4- and 5-speaker tests and 6.5/7.92 dB SISDRi on Mandarin/Japanese accompany positive-to-negative SISDRi as a speaker leaves the target conversation.Source: §4.3 · p. 6, Figure 3 · p. 6, Table 2 · p. 7

4.4 Subjective human evaluation

  • The overall mean opinion score rose from 1.88 to 4.30 after 11 participants compared the model output with original mixtures.Source: §4.4 · p. 7, Table 3 · p. 7

4.5 Ablation studies

  • 12.48 dB versus 1.45 dB SISDR improvement favored the dual model over the single streaming model; 4-second updates and speaker embeddings reduced SISDRi by 1.22 and 2.65 dB.Beamformer self-speech versus ground-truth self-speech differed by less than 0.38 dB.Source: §4.5 · pp. 7–8
  • Turn-taking disruption degraded performance, while 10-, 5-, and 1-second contexts reduced SISDRi by 2.12, 4.06, and 5.74 dB versus full context.Source: Table 5 · p. 8, §4.5 · p. 8

5 Real-World Egocentric Recordings

The real-world evaluation tests egocentric recordings and constructed mixtures, then examines generalization, partner pickup, turn-change collisions, and prolonged wearer silence. Results show that performance depends on conversational dynamics, including the wearer’s self-speech anchor.

5 Real-World Egocentric Recordings

  • The real-world dataset contains 6.8 hours of binaural egocentric recordings from 11 participants, including seven two-speaker and five three-speaker conversations.Open-ended English discussions took place amid typical background noise.Source: §5 · p. 8

5.1 Data pre-processing

  • Mixtures pair a target conversation with a separate interfering conversation, begin with at least 3 seconds of wearer self-speech, and use target SNRs sampled from –10 to 10 dB.Interference involves two speakers, while the target has two or three; 200 such mixtures form an out-of-distribution test set.Source: §5.1 · p. 9

5.2 Real-world Results

  • Three-speaker performance drops because its turn-taking dynamics were synthesized in training; fine-tuning on real three-speaker conversations may improve results.The evaluation also reports real-world generalization from simulated training data, with augmentation improving performance by 1.73 dB.Source: §5.2 · p. 9
  • After partners begin speaking, SISDRi is 4.77 dB in the first 0–2 seconds and exceeds 8 dB after 2 seconds.Pickup was assessed with a 2-second sliding window over non-wearer turns.Source: §5.2 · p. 9
  • For 11.3% of turns, target and interfering turn changes are less than 1 second apart, and SISDRi falls to 4.98 dB.The authors suggest closely timed changes can confuse the model and identify conversation content as a possible future direction.Source: §5.2 · p. 9
  • When the wearer remains silent for over 2 minutes, partner SISDRi drops below zero and recovers after the wearer resumes speaking.During intermittent wearer speech, SISDRi stayed above 5 dB.Source: §5.2 · p. 9, Figure 5 · p. 9

6 Conclusion

The conclusion presents a real-time, on-device assistant that identifies the wearer’s conversation partners and suppresses unrelated speech without explicit prompts. It generalizes to real-world egocentric recordings despite synthetic-only training, using turn-taking to model engagement.

6 Conclusion

  • The assistant automatically identifies the wearer’s conversational partners and suppresses unrelated speech without explicit prompts.The conclusion describes it as real-time and on-device.Source: §6 · p. 9
  • The system generalizes to real-world egocentric recordings despite being trained only on synthetic data.Source: §6 · p. 9
  • By leveraging turn-taking to model conversational engagement, the approach connects speech separation with dialogue modeling.The authors describe this as a step toward assistants that interpret and adapt to conversation dynamics.Source: §6 · p. 9

7 Limitations and risks

The system is designed for active wearer participation and may have difficulty with overlapping speakers, especially simultaneous starts. The authors also note possible gains from real-world and language- or culture-specific adaptation, alongside risks from incorrect detection and the need for user controls.

7 Limitations and risks

  • The assistant is designed for active participation and is not suited to passive listening, including eavesdropping or passive consumption.Source: §7 · p. 10
  • Prioritizing real-time, on-device operation and turn-taking may limit disambiguation when multiple speakers begin simultaneously.Lightweight content-aware models are proposed as a future direction.Source: §7 · p. 10
  • Performance may benefit from supervised adaptation to real-world acoustic and conversational conditions.Source: §7 · p. 10
  • Cultural and linguistic differences in turn-taking may make language- or culture-specific fine-tuning useful for robustness.The paper reports evaluations on English, Mandarin, and Japanese datasets.Source: §7 · p. 10
  • Incorrect detection may suppress relevant voices or amplify unrelated ones, a particular concern in high-stakes or fast-moving conversations.The authors recommend clear user means to override or adjust system behavior.Source: §7 · p. 10

A Dual Model architecture details

The dual-model architecture pairs a streaming extractor with a slower conversation-embedding model, combining low-latency processing with longer-range conversational context. Both models operate on time-frequency audio representations, while a separate binaural beamformer extracts the wearer’s self-speech.

A Dual Model architecture details

  • The streaming model processes each arriving chunk with low latency, conditioned by a slower model’s conversation embedding that captures longer-term dynamics.The slow model buffers T seconds of audio and updates the embedding used by the streaming model for the next T seconds.Source: A Dual Model architecture details · p. 13
  • Both models convert audio to a time-frequency representation with an STFT and concatenate its real and imaginary components along the channel dimension.The resulting tensor is provided as model input.Source: A Dual Model architecture details · p. 13
  • The conversation-embedding model uses chunk-local bidirectional LSTMs and causal attention across pooled chunk representations to model longer-term relationships.Its output is shifted by T seconds to simulate the slow model’s algorithmic latency before conditioning the streaming model.Source: A Dual Model architecture details · p. 13
  • The streaming model uses a unidirectional temporal LSTM and omits the global module, then reconstructs output audio through deconvolution and inverse STFT.Its conversation embedding is multiplied element-wise with the feature map between the first and second extraction blocks.Source: A Dual Model architecture details · p. 13

B Beamformer model details

B.1 Beamformer datasets

The beamformer is trained first on synthetic binaural mixtures and then fine-tuned on real-world recordings, addressing its sensitivity to spatial cues. Its evaluation uses unseen participants and rooms, and reports noise reduction and self-speech extraction.

B.1 Beamformer datasets

  • The beamformer uses binaural headset audio and spatial cues, including inter-channel phase and level differences, to extract the wearer’s speech amid speech and noise.The passage notes these spatial features are difficult to model accurately in simulation.Source: §B.1 · p. 14
  • The beamformer is pretrained on synthetic binaural recordings and fine-tuned on a smaller real-world set to generalize to real-world acoustic conditions.Synthetic mixtures combine wearer speech with one to five interfering speakers; real-world fine-tuning uses recordings collected across participants and rooms.Source: §B.1 · p. 14

B.2 Beamformer training

  • Training uses two stages, with negative SNR loss for synthetic pretraining and a composite L1 and multiresolution STFT loss for real-world fine-tuning.The fine-tuning loss weights spectral convergence and log magnitude by 1 each and linear magnitude by 4.Source: §B.2 · pp. 14–15

B.3 Beamformer real-world evaluation

  • Real-world evaluation uses recordings from six unseen participants in three unseen rooms, mixed at SNRs uniformly sampled from −5 to 5 dB.The evaluation mixtures pair self-speech with interfering speech recorded by the same participant.Source: §B.3 · p. 15
  • The unseen-real-world evaluation reports significant noise reduction and self-speech extraction by the beamformer.Table 8 covers evaluation on unseen real-world mixtures; its caption identifies DNSMOS BAK as an estimate of background-noise quality.Source: §B.3 · p. 15, Table 8 · p. 15

C Datasets

  • Except for Libri (leaving), the described datasets use two-speaker interference conversations with no speaker shared between target and interference conversations.The passage introduces dataset generation and spatialization as the subsequent process.Source: C Datasets · p. 15

C.1 Dataset Generation

The dataset-generation procedures create mixtures of target and interference conversations, including varying target-speaker counts and a scenario where a partner leaves. Synthetic egocentric audio is spatialized through room impulse responses to microphones at the wearer’s ears.

C.1 Dataset Generation

  • Libri mixtures combine target and interference conversations, with 200 test samples each for 2-, 3-, 4-, and 5-speaker targets and a leaving-speaker condition.The dataset has 16,000 training, 2,600 validation, and 1,000 test samples; target input SNR is sampled uniformly from −10 to 10 dB.Source: §C.1 · p. 15
  • Libri target conversations begin with at least five seconds of wearer speech, and each target partner must contribute an utterance longer than five seconds.For Libri (2 spk), the other target speaker has a minimum total utterance duration of five seconds; Libri (3 spk) requires both partners to have a qualifying utterance.Source: §C.1 · p. 15
  • Libri (leaving) models a partner moving from a three-speaker target conversation into the interference conversation, leaving two target speakers.The transition is placed before the 40-second mark, and the leaving speaker must have a consecutive five-second utterance in the interference conversation.Source: §C.1 · p. 16

C.2 Dataset Spatialization

  • Synthetic egocentric audio is generated with room impulse responses from each speaker, including the wearer, to microphones positioned at the wearer’s ears.Room length and width range from 5 to 10 m, height from 3 to 4 m, and RT60 from 0.15 to 1 s.Source: §C.2 · p. 16

D Training details

Training proceeds through joint model training on conversational data, spatialized-data training, and a final stage that varies utterance timing. The runtime analysis reports inference-time distributions, while a waveform example illustrates extraction of a quieter conversation partner.

D Training details

  • The first two stages jointly train the slow and fast models for 120 and 50 epochs, respectively, with the slow model initialized from first-stage pretrained weights in stage two.Stage one uses selected Libri and Candor mixtures; stage two uses the spatialized dataset.Source: D Training details · p. 16
  • The final training stage randomizes the silence between successive partner utterances, often shifting them into overlap with the wearer’s speech.It runs for 42 epochs with batch size 4 on two A100 GPUs.Source: D Training details · pp. 16–17

E Runtime analysis

  • Figure 6 reports inference-time cumulative distribution plots for the fast model processing 12.5 ms chunks on OrangePi 5B and the slow model processing 1 s chunks on Apple Silicon M2.The figure compares each model’s inference time on its stated device and chunk size.Source: Figure 6 · p. 17, E Runtime analysis · p. 17

F Conversation waveform examples

  • In the spatialized Libri (2 spk) example, the model captures the other target speaker’s low-amplitude speech despite the wearer’s beamformed self-speech being louder.Spatialization makes the self-speech sound closer to the wearer’s ears and higher in amplitude than the other target speaker.Source: F Conversation waveform examples · p. 17

G Details of Ablation studies

The appendix details ablation protocols and the user-study design, then describes the participants and conversational statistics used for real-egocentric recordings.

G Details of Ablation studies

H User Study Design

  • Participants rated noise suppression, conversation comprehension, effort, and overall experience after hearing both the original mixture and model output, using 5-point scales.Source: H User Study Design · p. 18

I Egocentric evaluation participants

J Real-egocentric conversation analysis

Loading 2511.11473v1…