PaperlayerView exact revision on arXiv

Source-linked AI summary

GLARE: Generating Listening Heads with Appropriate Reactions

Zikai Liao, Yumin Suh, Yi Ouyang, Yi-Lun Lee, Yi-Hsuan Tsai, Zhaozheng Yin

arXiv:2609.40317v1cs.CV

TL;DR

Fine-grained listener-reaction annotations are limited in dyadic datasets, and common visual-generation metrics do not evaluate reaction appropriateness. The paper curates an annotated dataset, proposes GLARE and reaction-centric metrics, and reports gains in visual and reaction-related measures on RealTalk and Seamless.

  • Problem

    Existing dyadic datasets lack event-level reaction type, timing, and duration annotations, while conventional metrics do not assess reaction appropriateness and alignment.

    Source: §1 · pp. 1–2

  • Method

    The paper curates aligned speaker-listener clips with event-level annotations, introduces prosody-conditioned GLARE with temporal reaction supervision, and proposes reaction-centric evaluation metrics.

    Source: §1 · p. 2, §5 · p. 6

  • Results

    GLARE achieves the best overall performance across most metrics on RealTalk and maintains strong visual fidelity while improving reaction-related metrics on Seamless.

    Source: §6.2 · p. 7

  • Takeaways & Limitations

    Reaction occurrence, temporal alignment, asymmetric timing deviation, and reaction-region visual quality can be assessed alongside conventional generation measures.

    Source: §5 · p. 6

    Applying reinforcement learning would require a reward function that accurately reflects human judgments of reaction appropriateness.

    Source: F Discussion of Limitations and Future… · p. 23

Abstract

from arXiv · show

While talking head generation has advanced rapidly, generating natural listener behavior in dyadic conversations, which know when to react, how to react, and with what type of response, remains underexplored. Existing dyadic datasets lack fine-grained listener reaction annotations, and prevailing evaluation metrics inherited from talking-head and video generation measure visual realism rather than whether a listener reacted appropriately. We address these gaps along three aspects. First, we curate a listening-head-specific dataset built from RealTalk and Seamless Interaction, comprising approximately 147 hours of paired speaker-listener videos with 64,557 event-level reaction annotations across six categories: nodding, head shaking, smiling, laughing, frowning, and surprised. Second, we introduce an audio-driven baseline built on a flow-matching transformer, namely GLARE, with prosody conditioning derived from Qwen2-Audio and a temporal reaction loss that explicitly supervises frame-wise reactions. Third, we propose a reaction-oriented evaluation protocol that jointly measures reaction occurrence (R-F1), temporal alignment (R-tIoU), asymmetric temporal deviation (R-ATD), and reaction-region visual quality (R-FID), giving a more behaviorally grounded assessment than visual-quality-only metrics. Experiment results show consistent gains over prior listening-head methods in both visual fidelity and reaction-level metrics, suggesting that reaction-aware data, modeling, and evaluation are critical for natural listening behavior.

1 Introduction

Listening-head generation lacks fine-grained reaction supervision, while common visual-generation metrics do not assess whether reactions are appropriate. The paper addresses these gaps with an annotated dataset, GLARE, and reaction-oriented evaluation.

1 Introduction

  • Listener responses such as smiling and nodding convey attention, agreement, hesitation, and affective engagement.Visually plausible motion can still appear unnatural when reactions are absent, mistimed, or inconsistent with context.Source: §1 · p. 1
  • Existing dyadic datasets lack event-level supervision for reaction type, timing, and duration, limiting fine-grained modeling of listener responses.ViCo provides task-specific supervision but is limited in scale; larger resources lack event-level reaction annotations.Source: §1 · p. 1
  • The curated listening-head dataset contains approximately 147 hours of video and 64,557 event-level reaction instances.It is built from aligned speaker-listener video-audio pairs.Source: §1 · p. 2
  • GLARE is an audio-driven listening-head generator equipped with prosody conditioning and a temporal reaction loss.These components target speaker-side cues and reaction timing.Source: §1 · p. 2
  • The proposed reaction-oriented evaluation jointly assesses visual fidelity, reaction accuracy, and temporal alignment.Existing visual metrics do not explicitly assess reaction type consistency, alignment with conversational cues, or reaction-segment quality.Source: §1 · p. 2

2 Related Works

Prior listening-head work has advanced listener synthesis, but most methods do not model reactions as categorized, temporally localized events. Related datasets lack fine-grained reaction annotations, and existing evaluation measures do not jointly assess reaction type, timing, and reaction-specific visual quality.

Listening head generation

  • Most listening-head methods optimize motion realism, diversity, or synchrony rather than categorized, temporally localized listener reactions.Prior work includes non-deterministic motion, emotion conditioning, diffusion-based generation, dyadic modeling, and real-time interaction.Source: §2 · p. 2

Interactive conversation datasets

  • Larger dyadic resources such as RealTalk, Seamless, and SpeakerVid-5M provide conversational videos but lack fine-grained listener reaction events.The proposed dataset instead aligns speaker-listener clips with event-level reaction type, timing, and duration annotations.Source: §2 · p. 2

Evaluation methods

  • Common evaluation measures feedback timing or global visual quality, but do not jointly assess reaction type, timing, and reaction-specific motion quality.The proposed evaluation treats reactions as categorized temporal events and measures occurrence, alignment, asymmetric timing deviation, and reaction-region quality.Source: §2 · p. 2
  • The data-curation pipeline progresses from conversational videos through facial cropping and speaker-listener pairing to frame-wise reaction annotations.Audio separation and diarization identify timed listening and speaking segments before clips are paired.Source: Figure 1 · p. 3

3 Dataset Curation

The dataset pipeline converts dyadic conversations into aligned speaker-audio and listener-video clips, then adds categorized temporal reaction annotations. The resulting resource contains approximately 147 hours of data and 64,557 reaction instances.

3 Dataset Curation

  • The curated resource contains 107,149 speaker-audio/listener-video pairs, corresponding to approximately 147 hours of data.It is built from RealTalk and Seamless Interaction.Source: §3 · p. 3

3.1 Speaker–Listener Clip Construction

  • Visual filtering removes unsuitable videos and face crops before portrait-centered listener and speaker clips are extracted.Speech activity and diarization are used to pair active-speaker audio with the synchronized portrait video of the other participant.Source: §3.1 · p. 3

3.2 Reaction Taxonomy and Annotation

  • The reaction taxonomy covers six visually recognizable categories, and manual verification yields 64,557 reaction instances.The categories are nodding, head shaking, smiling, laughing, frowning, and surprised; annotations provide event boundaries and frame-wise scores for evaluation and training.Source: §3.2 · p. 4

4 Methodology of GLARE

GLARE uses a conditional flow-matching transformer to generate listener motion latents from speaker audio, listener identity, and temporal context. It adds prosody conditioning and frame-wise reaction supervision, then decodes predicted latents into listener video.

4.1 Preliminaries

  • GLARE conditions a DiT-style flow-matching transformer on listener motion and identity, speaker audio, and emotion features to predict listener motion latents.The predicted vector fields are solved into latents that are combined with listener identity and decoded into frames.Source: §4.1 · p. 4

4.2 Prosody Conditioning

  • A prosody branch converts frame-aligned speaker prosodic intensity into a latent representation and concatenates it with the other driving conditions.The prosody signal is derived from timestamps of prosodic fluctuations and projected through an MLP and Sigmoid layer.Source: §4.2 · pp. 4–5

4.3 Temporal Reaction Loss

  • The temporal reaction loss supervises six class-specific frame-wise reaction predictions against annotated reaction scores.It is intended to preserve reaction-aware temporal structure and improve reaction timing and type consistency.Source: §4.3 · p. 5

4.4 Training Objective and Inference

  • Training combines flow-matching, temporal velocity-consistency, and reaction losses; inference integrates predicted vector fields with an ODE solver to obtain motion latents.The resulting latents are decoded into listener video frames.Source: §4.4 · p. 6

5 Reaction-Centric Evaluation Metrics

The section introduces reaction-centric metrics for assessing whether listener reactions are correctly typed, temporally aligned, and visually realistic. The metrics provide complementary measurements, but compare generated behavior against ground truth rather than absolute plausibility.

5 Reaction-Centric Evaluation Metrics

  • Predicted reactions match ground-truth events only when types agree and temporal IoU meets τ = 0.5; one-to-one matching produces event pairs for scoring.Source: §5 · p. 6
  • R-F1 measures reaction occurrence and type consistency from event-level matches, while R-tIoU averages temporal IoU across matched events.Source: §5 · p. 6
  • R-ATD combines normalized start-time, end-time, and duration deviations, penalizing premature or truncated reactions more heavily than slightly delayed or extended ones.Source: §5 · pp. 6–7
  • R-FID measures visual quality using frames inside predicted reaction segments and ground-truth reaction intervals, rather than all frames dominated by neutral listening.Source: §5 · p. 7
  • Together, the four metrics assess reaction correctness, temporal localization, asymmetric timing deviation, and reaction-region realism, but ground truth is a reference rather than an absolute plausibility measure.Source: §5 · p. 7

6 Experiments

Experiments compare GLARE with prior methods on two datasets, test human judgments, and examine component and loss-weight ablations. GLARE improves reaction modeling alongside visual quality, while results vary by dataset, reaction type, and supervision balance.

6.1 Experiment Settings

  • Experiments use a 9:1 train/test split and compare L2L, DIM, ViCo, ListenFormer, and DyStream, evaluating visual quality, motion, coordination, and reaction metrics.The generator uses a latent flow-matching transformer, with 25 FPS video and 16 kHz audio; pretrained motion and audio models are frozen.Source: §6.1 · p. 7

6.2 Experiment Results

  • On RealTalk, GLARE achieves the best overall performance across most metrics; on Seamless, it maintains strong visual fidelity and improves reaction metrics, especially R-F1, R-tIoU, and R-FID.DyStream obtains slightly higher Var, while GLARE provides a better balance between motion diversity, reaction accuracy, and visual quality.Source: §6.2 · p. 7, Table 2 · p. 8
  • Per-class results show laughing is easier to detect and align, nodding and head shaking are timing-sensitive, and surprised is most challenging.Smiling has more balanced performance, while Seamless scores consistently lower than RealTalk, reflecting greater diversity and complexity.Source: §6.2 · p. 7, Table 3 · p. 8
  • Human evaluation gives GLARE 89.9% HAR for naturalness, 89.4% for contextual appropriateness, and 93.5% for timing plausibility.More expressive reactions remain challenging; laughter's contextual appropriateness depends on subtle semantic and social cues.Source: §6.2 · p. 8, Table 4 · p. 9
  • On RealTalk, GLARE produces more expressive, contextually consistent reactions that better match ground truth in type and timing than prior methods.L2L, DIM, and ViCo often produce static listeners, while ListenFormer and DyStream often mistime reactions.Source: §6.2 · p. 8, Figure 3 · p. 9

6.3 Ablation Study

  • Combining prosody conditioning with temporal reaction loss achieves the best overall performance; reaction loss substantially improves DI-Sync, R-F1, and R-tIoU while reducing R-ATD and R-FID.Prosody improves visual quality and motion dynamics but brings limited reaction-metric gains and can degrade temporal precision on its own.Source: §6.3 · pp. 8–9, Table 5 · p. 10
  • Increasing λreact from 0.01 to 0.05–0.1 improves visual and reaction metrics, but values ≥0.2 degrade both; λreact=0.05 achieves the best trade-off.The results indicate that overly strong reaction supervision disrupts the balance between generation quality and reaction accuracy.Source: §6.3 · p. 9, Table 5 · p. 10
  • Prosody conditioning improves R-F1 across reaction types, with the largest gain for laughing, but most categories show degraded R-tIoU and R-ATD.The authors note that prosody does not explicitly determine whether or when a listener should react, so temporal reaction loss supplies explicit occurrence and timing supervision.Source: Table 6 · p. 10, §6.3 · p. 10

7 Conclusion

The conclusion presents a reaction-centric framework for listening-head generation, combining a dataset, an audio-driven baseline, and reaction-oriented metrics. Experiments on RealTalk and Seamless show improvements over existing approaches, especially in reaction accuracy and temporal alignment.

7 Conclusion

  • The framework combines aligned speaker-listener pairs with fine-grained event annotations, prosody-conditioned audio-driven generation, temporal reaction loss, and metrics for reaction occurrence, alignment, and visual quality.The dataset draws on RealTalk and Seamless.Source: §7 · p. 10
  • Experiments on RealTalk and Seamless improve visual fidelity, motion naturalness, reaction accuracy, and temporal alignment over existing approaches.Source: §7 · p. 10
  • The authors conclude that explicit reaction supervision and reaction-specific evaluation are important steps toward more natural, context-aware listening-head generation.Source: §7 · p. 10

A.1 Raw Data Sources

The authors curate dyadic videos into aligned speaker-audio/listener-video pairs and annotate listener reactions as temporal events. The pipeline filters videos, processes audio, constructs reaction labels, and manually verifies annotations.

A.1 Raw Data Sources

  • RealTalk and Seamless Interaction provide the conversational videos, which are processed into speaker-audio/listener-video pairs with event-level listener-reaction annotations.Seamless Interaction offers a larger collection of dyadic audiovisual recordings with high-resolution video and cleaner audio.Source: §A.1 · p. 14

A.2 Video Filtering and Face-Crop Quality Control

  • Videos with scene changes, artifacts, missing faces, occlusions, unstable views, or excessive participant overlap are discarded, and participant portraits are cropped and quality-checked.The face box is enlarged by 1.5 to retain contextual head motion; crops with unreliable faces or non-frontal views are removed.Source: §A.2 · p. 14

A.3 Audio Processing and Speaker–Listener Pairing

  • RealTalk audio is separated, denoised, and diarized, while Seamless audio and diarization are used when reliable, with timestamp correction as needed.Speech-activity timelines define speaker-dominant intervals, pairing each active speaker’s audio with the synchronized video of the other participant.Source: §A.3 · p. 14

A.4 Clip-Level Filtering

  • Clips with invalid or very short speaker audio, missing listener frames, unreliable crops, or unclear listener faces are removed before alignment at 25 FPS and 16 kHz.The filtering retains samples with a valid speaker signal and stable listener portrait sequence.Source: §A.4 · p. 15

A.5 Data License, Privacy, and Release Policy

  • The release provides the annotation pipeline and processed dataset, but not raw media, frames, cropped faces, transcripts, or media segments.The authors rely on the original datasets’ consent and release procedures and report using each under its stated terms.Source: §A.5 · p. 15

B.1 Frame-Wise Reaction Representation

  • The detector converts frame-wise confidence scores for six reaction classes into temporal events by thresholding, extracting connected components, and removing or merging short detections.Overlapping events of different classes are resolved by retaining the higher-confidence event; ambiguous overlaps are removed during manual verification.Source: §B.1 · p. 15

B.2 Head-Motion-Based Reactions

  • Nods are detected from normalized vertical head motion, while head shaking uses normalized horizontal motion and yaw-related dynamics.Nod candidates must show upward and downward extrema, satisfy duration constraints, and exceed an amplitude floor; retained intervals produce dense confidence scores.Source: §B.2 · pp. 15–16

B.3 Expression-Based Reactions; B.4 Manual Verification

  • Facial-expression reactions are detected from landmark dynamics, action units, and emotion predictions, then manually checked to remove weak, ambiguous, or spurious events.Temporal smoothing reduces single-frame false positives, and cross-class overlaps are resolved to keep annotations single-label and non-overlapping.Source: §B.3 · pp. 16–17, §B.4 · p. 17

C Model and Training Details

The baseline adapts FLOAT to listener generation using reference and prior-motion context, with training loss applied only to generated frames. Prosody cues are extracted from speaker audio, aligned to video frames, and incorporated into the conditioning features.

C.1 Architecture and Optimization

  • The FLOAT-based baseline adds a reference clip alongside previous motion frames because speaker audio alone produced overly static listener motion in preliminary experiments.Training orders the reference before prior and current motion, computes loss only on generated frames, and uses one fixed reference clip at inference.Source: §C.1 · p. 17

C.2 Prosody Extraction and Frame Alignment

  • Qwen2-Audio-7B-Instruct detects speaker-side localized prosodic fluctuations as temporal intervals with confidence scores; sparse intervals become a frame-aligned scalar prosody sequence at 25 fps, then an MLP and sigmoid project it.Source: §C.2 · pp. 17–18

D Evaluation Protocol Details

The protocol extracts generated and annotated reaction events, matches them class-wise, and aggregates event metrics. It also applies asymmetric timing penalties and computes visual quality on reaction-region frames.

D.1 Event Extraction and Time Representation

  • Generated reaction events are extracted with the same detector and post-processing as the ground truth, and intervals are represented as half-open spans at 25 FPS.All methods use the same thresholds and matching protocol.Source: §D.1 · p. 19

D.2 One-to-One Class-Aware Matching

  • Events are matched independently by reaction class using descending temporal IoU and greedy one-to-one assignment above a threshold.This prevents one prediction from matching multiple ground-truth events, or vice versa.Source: §D.2 · p. 19

D.3 Event-Level Aggregation

  • R-F1 aggregates true positives, false positives, and false negatives, while R-tIoU and R-ATD are computed over matched pairs.When no matched pair exists, R-tIoU is set to zero and R-ATD is not reported; this does not occur in the reported experiments.Source: §D.3 · p. 19

D.4 R-ATD Penalization

  • R-ATD penalizes premature starts, early endings, and reactions shorter than ground truth more strongly, using α = 2 and β = 0.5.The symmetric α = 1, β = 1 setting yields a lower numerical R-ATD on RealTalk, but omits those additional penalties.Source: §D.4 · p. 20, Table 10 · p. 20

D.5 R-FID Computation

  • R-FID compares visual features from generated frames inside predicted reaction intervals with features from ground-truth reaction frames.Overlapping intervals count each frame once, and reaction frames are pooled at dataset level because reactions are sparse.Source: §D.5 · p. 20

E.1 Qualitative Results on Seamless Dataset

On Seamless, qualitative examples show GLARE producing more expressive and better-aligned reactions than comparison methods. Ablations report effects of audio context length, prosody projection size, and reaction-loss choice.

E.1 Qualitative Results on Seamless Dataset

  • GLARE produces more visible and contextually appropriate reactions, including a well-timed nod to speech emphasis and a subtle smile better synchronized with prosodic cues.The qualitative comparison reports improved reaction type correctness and temporal alignment over prior methods.Source: §E.1 · pp. 20–21, Figure 4 · p. 21

E.2 More Ablation Studies

  • 10s of speaker audio achieves the best balance: increasing context from 2s improves most reported metrics, while 20s or 50s brings marginal gains or slight degradation.The authors attribute the latter pattern as likely related to temporal redundancy and modeling difficulty.Source: §E.2 · p. 22, Table 11 · p. 22
  • Projecting prosody from one channel to four yields the best trade-off, while eight channels gives no consistent gains and slightly degrades several metrics.The reported gains from four channels span generation quality and reaction-related measures.Source: §E.2 · p. 22, Table 12 · p. 22
  • Smooth-L1 improves generation quality and temporal-deviation metrics over BCE, while BCE scores slightly better on DI-Sync, R-F1, and R-tIoU.Smooth-L1 is adopted as the default reaction loss.Source: §E.2 · p. 22, Table 13 · p. 22

F Discussion of Limitations and Future Works

The discussion identifies limits in reaction evaluation, modeled behavior, input scope, and conversational settings. It also notes potential applications alongside risks from realistic synthetic listener behavior.

F Discussion of Limitations and Future Works

  • R-F1, R-tIoU, R-ATD, and R-FID measure consistency with one observed human reaction, not whether a reaction is absolutely socially appropriate.Different listeners may respond differently to the same cue in type, timing, or intensity.Source: F Discussion of Limitations and Future… · p. 23
  • The framework models explicit reactions such as nodding and smiling, but not subtler gaze or facial changes or baseline motion and posture drift.These more continuous behaviors are harder to annotate.Source: F Discussion of Limitations and Future… · p. 23
  • The current model only partially captures language semantics, personality, and conversational context, which also influence listener behavior.The authors identify richer multimodal understanding and more comprehensive motion modeling as future work.Source: F Discussion of Limitations and Future… · p. 23
  • Supervised temporal reaction modeling uses frame-level annotations, while reinforcement learning would require a reward reflecting human judgments of appropriateness.The authors suggest preference learning may further improve generation quality.Source: F Discussion of Limitations and Future… · p. 23
  • GLARE targets conversations with one speaker, not full-duplex or multi-party settings; proposed extensions use speaking/listening switching or interaction-level state arbitration.The arbitrator would determine who is speaking or listening and when states change.Source: F Discussion of Limitations and Future… · p. 23

G Broader Impact

  • Potential applications include virtual assistants, social robots, telepresence, and human-computer interaction, where responsive listener reactions may support more engaging interactions.The authors frame these as potential applications and effects.Source: G Broader Impact · p. 23
  • Realistic listener behavior may enable misleading or deceptive content, and reactions conveying agreement or emotional alignment may be misinterpreted or misused.The authors call for disclosure of synthetic content, licensing compliance, and attention to fairness and diversity.Source: G Broader Impact · p. 23
Loading 2609.40317v1…