Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videos

1 Introduction

• Offline clip-level labels poorly capture short-window state changes, while a single negative class merges background, self-talking, and talking-to-others despite their different recognition challenges.Talking-to-others includes genuine speech and visible facial motion, whereas self-talking places the wearer in the speaker role.Source: §1 · p. 1
• The paper reformulates TTM as online frame-level prediction and preserves multiple socially meaningful non-TTM states rather than collapsing them into one negative class.The Online TTM Dataset contains approximately 900K frames derived from Ego4D, with reconstructed frame-level supervision and temporally coherent re-sampling to mitigate class imbalance.Source: §1 · p. 2
• The benchmark is cleaned to remove missing-frame cases, annotation-coverage issues, and samples with prolonged absence of valid face boxes.Its construction preserves temporal context around TTM events, segment continuity, and informative state boundaries.Source: §1 · p. 2
• The online model combines facial cues, wearer-centered geometry, acoustic features, and Whisper-derived speech semantics to distinguish speaking states.Its streaming predictions use the current frame and recent history without future frames.Source: §1 · p. 2
• 75.5% frame-level F1 on TTM; the proposed model outperforms strong baselines across frame-level F1 and segment-level mAP.Semantically enhanced audio representations are more effective than conventional acoustic features alone, and four-state modeling supports analysis of socially confusable cases.Source: §1 · p. 2

2.1 Egocentric social interaction

• Ego4D established wearer-centered Looking-at-Me and Talking-to-Me tasks; later work explores cross-task translation, audio-visual fusion, social context, conversational relations, and robustness.CASTLE and its challenge support synchronized ego- and exo-centric multimodal analysis but do not specifically address frame-level wearer-directed speaking-state recognition.Source: §2.1 · pp. 2–3
• The illustrated Ego4D clips contrast long background intervals and sparse TTM segments with frequent speaking-state transitions and unlabeled gaps.These examples show variation in the original social annotations across clips.Source: Figure 2 · p. 3

2.2 Active Speaker Detection

• ASD research developed frame-level benchmarks and methods for temporal reasoning, multimodal fusion, and causal streaming, but ASD detects visible speakers rather than wearer-directed speech.The paper adapts ASD methods to TTM using a structured four-state prediction framework.Source: §2.2 · p. 3

3.1 Problem Definition

• TTM is reformulated as a frame-level task that predicts the current speaking state from available observations rather than assigning one label to an entire segment.The model may use current and past observations, but not future frames.Source: §3.1 · p. 3

3.2 Challenges of Online Egocentric TTM

• Speaking states can change within short intervals, so online TTM depends on when a person turns from talking to others toward the wearer.The wearer may also begin speaking during an ongoing interaction.Source: §3.2 · p. 4
• TTO and SELF can resemble TTM because they may share genuine speech, visible lip movements, salient audio, or wearer-centered attention cues.These overlaps make the task harder than speech-versus-background detection.Source: §3.2 · p. 4
• The online causal constraint excludes future frames, preventing the model from using later evidence to resolve ambiguous moments.This setting is described as more consistent with wearable assistants, augmented reality, and interactive egocentric perception.Source: §3.2 · p. 4

3.3 Dataset Construction and Statistics

• Frame-level supervision is reconstructed from segment labels, then task-aware resampling retains all TTM frames and informative nearby BG, SELF, and TTO cases.TTM accounts for 11.49% of usable frames in the original distribution.Source: §3.3 · p. 4
• About 40% of sampled windows contain at least two speaking states, preserving transitions such as TTO-to-TTM and SELF-to-BG.The cleaning procedure removes samples with missing frames or abnormal annotation coverage, and sequences with over 40% black or no-face frames.Source: §3.3 · p. 4
• 885,866 frame-level samples from 406 videos form the Online TTM Dataset, split 334/21/51 across training, validation, and testing.The benchmark includes 367,529 TTM, 100,321 SELF, 170,321 TTO, and 247,695 BG frames.Source: §3.3 · pp. 4–5
• The model combines person-centered visual cues with aligned audio-text evidence, fuses them by cross-attention, and predicts four states using a causal temporal Transformer.The visual cues include facial appearance, head-related representation, mouth motion, and wearer-centered geometry; the audio-text branch includes acoustic evidence and Whisper-derived speech semantics.Source: Figure 3 · p. 5

4.1 Overview

• The model fuses person-centered visual and audio-text features, then uses a causal temporal Transformer to predict four speaking states at each frame.Visual features query audio features in cross-attention; text semantics are added to the fused representation before temporal modeling.Source: §4.1 · p. 5, §4.4 · p. 6

4.2 Visual Encoder

• The visual encoder combines facial appearance, head information, mouth motion, and wearer-centered geometry to represent who is visible and whether interaction is directed toward the wearer.The geometry branch uses face-box center, area, and aspect ratio as spatial cues, particularly in multi-person scenes.Source: §4.2 · pp. 5–6

4.3 Audio and Speech-Semantic Encoding

• The audio-text branch pairs frame-synchronous acoustic features with Whisper-derived speech semantics aligned to each utterance’s temporal span.Acoustic features capture local speech patterns, while transcript-derived semantics provide an additional cue for socially grounded TTM detection.Source: §4.3 · p. 6

4.4 Online Multimodal Fusion and Causal Temporal Modeling

• Visual-query, audio-value cross-attention and a causal temporal Transformer integrate modalities using current and recent past observations only.The windowed causal mask uses a past context length of 32 frames and excludes future frames.Source: §4.4 · p. 6

4.5 Training Objective

• Training minimizes frame-level cross-entropy over valid, non-padded frames under the four-state formulation.Padded positions are ignored in the loss and attention masks, and label smoothing is applied during training.Source: §4.5 · pp. 6–7

5.1 Experimental Setup

• All baselines use the same four-state labels, causal online setting, and train/validation/test split; evaluation reports per-class frame-level F1 and TTM segment mAP.The comparison includes the Ego4D baseline, Streaming ASD, CMeRT, LoCoNet, and ProTAS.Source: §5.1 · p. 7, §5.2 · p. 7

5.2 Experimental Results

• 75.5% frame-level F1 on TTM and 79.8% TTM segment mAP; the method leads all four class-wise F1 metrics and improves over Streaming ASD by 9.2% TTM F1 and 7.0% segment mAP.Figure 4 compares predictions with ground truth on challenging scenarios and shows more accurate predictions in socially confusable cases.Source: §5.2 · p. 7, Figure 4 · p. 8
• 75.5% TTM F1 under four-state training compares with 64.2% under binary training, while offline inference reaches 87.2% versus 75.5% causally online.The comparison quantifies the performance cost of not using future observations.Source: §5.2 · p. 7

5.3 Ablation

• Adding Head improves BG, SELF, and TTM; Mouth further helps especially on SELF and TTO, while Geometry adds gains particularly on BG and TTO.The authors associate these improvements with head-related cues, mouth dynamics, and wearer-centered spatial priors, respectively.Source: §5.3 · p. 7
• 75.54% TTM frame-level F1 and 79.78% TTM segment mAP with text semantics compare with 65.00% and 72.21% without it; TTO F1 rises from 26.07% to 40.19%.The text-semantic branch produces substantial improvements across classes, especially BG, TTO, and TTM.Source: Table 3 · p. 7, §5.3 · p. 8
• With zeroed audio or face input, TTM F1 is 72.8% and 69.4%; face-box dropout of 30% or 50% yields 69.8% and 67.2%.Together with the no-text result, these tests indicate that the model does not rely exclusively on any one modality.Source: §5.3 · p. 8
• Shuffling transcripts while preserving silent intervals lowers TTM F1 from 75.5% to 69.6%, while BG F1 changes from 99.42% to 98.8%.The authors associate BG improvement largely with speech-presence information and accurate TTM recognition with temporally aligned semantics.Source: §5.3 · p. 8

5.4 Error Attribution Analysis

• Adding text semantics reduces aggregated BG-related TTM confusion from 33,150 frames to 111, while remaining errors concentrate on TTO/TTM confusion: 21,493 and 14,187 frames, respectively.The remaining errors are between socially similar speaking states rather than background conditions.Source: §5.4 · p. 8, Table 4 · p. 8

• The Online TTM Dataset preserves transitions among background, self-talking, talking-to-others, and TTM across approximately 900K frame-level annotations.The online multimodal model integrates visual cues, wearer-centered geometry, acoustic features, and ASR-derived speech semantics.Source: §6 · p. 8