Source-linked AI summary
Controlling Backchannels in Streamable Full-duplex Models
Maike Züfle, Peter Polák, Sefik Emre Eskimez, Jan Niehues, Peter Bell, Ondřej Klejch
TL;DR
Full-duplex dialogue models rarely explicitly model backchannels, and existing evaluations often do not assess whether their timing matches human conversation. This paper adds a threshold-controlled onset head to models’ own hidden states and evaluates its timing and generated backchannels. Across two architectures, the approach improves backchannel frequency and timing, and human listeners rate PersonaPlex-BC backchannels on par with human ones.
Problem
Full-duplex backchannel evaluations often measure whether models backchannel at all rather than whether placement matches real human conversational timing.
Source: §1 · p. 1Method
A lightweight head predicts backchannel onset from full-duplex model hidden states and force-decodes a backchannel token when the probability crosses a tunable threshold.
Source: §1 · p. 1, §2.1 · p. 2Results
Across two architectures, explicit control improves backchannel frequency and timing, and PersonaPlex-BC backchannels are rated on par with human ones.
Source: §6 · p. 4, §5.3 · p. 4Takeaways & Limitations
Backchannel timing can be learned as a lightweight, pluggable control signal rather than left to emerge implicitly from conversational training.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Backchannels, brief acknowledgements like "uh-huh" produced while the other party may still be talking, are central to natural conversation, but full-duplex spoken dialogue models rarely model them explicitly. We introduce a lightweight backchannel head that predicts, from a full-duplex model's own hidden states, when a backchannel should begin. Once this probability crosses a tunable threshold, a backchannel is force-decoded. Attached to both a 7B (PersonaPlex) and a 1B (F-Actor) model, it generalizes across scale. Probing confirms the hidden states anticipate real human timing, and generation evaluation shows more frequent, better-timed backchannels. Human raters judge the resulting backchannels on par with real ones.
1. INTRODUCTION
Full-duplex models can backchannel while a partner is speaking, but existing evaluations often ask only whether they backchannel, not whether timing is appropriate. The paper introduces an explicit timing-control head and evaluates it against human timing and judgments of appropriateness.
1. INTRODUCTION
- Prior full-duplex models produce backchannels implicitly through conversational training, while reinforcement learning can shape them but is expensive and offers no controllable timing signal.Related approaches add VAP heads for turn-taking or probe representations without pretraining.Source: §1 · p. 1
- The paper identifies a gap in evaluating whether backchannels are appropriately placed against real human conversational timing, beyond whether models backchannel at all.It argues that models should explicitly predict when a backchannel should occur.Source: §1 · p. 1
- A lightweight head predicts backchannel onset from a full-duplex model’s hidden states and triggers force-decoding when its probability crosses a threshold.It is attached to 7B PersonaPlex and 1B F-Actor; F-Actor is adapted for streaming inference.Source: §1 · p. 1
- The evaluation combines probing against human timing annotations with human ratings of generated backchannel appropriateness.The generated backchannels are rated on par with human backchannels.Source: §1 · p. 1
- The contributions include a method shown across two architectures and model scales, a streaming F-Actor version, and openly released code and models.The evaluation pairs hidden-state probing against human timing data with a human appropriateness study.Source: §1 · p. 1
2. CONTROLLABLE BACKCHANNELING
The proposed head predicts frame-level backchannel appropriateness and provides a separate control over how frequently backchannels are emitted. It maps backbone hidden states to onset probabilities, then uses thresholded triggering to insert a token that conditions backchannel generation.
2. CONTROLLABLE BACKCHANNELING
- The design targets frame-wise prediction of moments when a human would backchannel, while allowing emission frequency to be adjusted independently of placement quality.This is the stated objective of the model-independent backchannel head.Source: §2 · p. 2
2.1. Backchannel Head
- A two-layer MLP with a sigmoid maps the backbone’s hidden state at each timestep to a backchannel-onset probability.The probability refers to onset at t + 1.Source: §2.1 · p. 2
- Focal Loss trains the head on eligible frames where the partner is speaking and the agent is silent, addressing onsets’ roughly 1% share of conversational frames.The loss uses α = 0.9 and γ = 2.0.Source: §2.1 · p. 2
2.2. Thresholding and Conditioned Text Generation
- At inference, crossing a threshold triggers force-decoding of a special [BC] token, after which autoregressive generation produces the backchannel’s lexical form.The [BC] token’s loss is masked so the backbone does not learn to emit it spontaneously.Source: §2.2 · p. 2
3. EXPERIMENTAL SETUP
The study attaches the backchannel mechanism to PersonaPlex and an adapted streaming version of F-Actor, then trains on Fisher conversations and evaluates on human-human TurnBench conversations. The supplied figures describe layerwise onset-timing F1 and predicted backchannel frequency on TurnBench.
3.1. Models
- PersonaPlex (7B) is trained with the backchannel mechanism, while F-Actor (1B) is adapted for streaming by replacing NanoCodec with causal Mimi and adding a frozen depth transformer.PersonaPlex uses LoRA fine-tuning; the F-Actor temporal transformer is Llama 3.2 1B Instruct.Source: §3.1 · p. 2
3.2. Data
- Training uses around 2000 hours of Fisher English conversations, augmented with backchannels at selected agent-silent intervals while the interlocutor continues speaking.Augmentation locations require at least 1.5 seconds of agent silence and at least 1 second of continued interlocutor speech.Source: §3.2 · p. 2
- TurnBench evaluation uses 12 conversations totaling 137 minutes, with 574 annotated backchannels among 1,888 utterances, and leave-one-conversation-out crossvalidation.Utterance-level timestamps are converted to word-level timestamps using Parakeet TDT.Source: §3.2 · p. 2
- The supplied figures present layerwise exact-frame onset F1 and predicted backchannel frequency for base and backchannel-head-trained models on TurnBench.The onset F1 caption counts a prediction only when it lands on the exact annotated frame.Source: Figure 1 · p. 3, Figure 2 · p. 3
4. EVALUATION
The evaluation probes whether hidden states predict human backchannel onsets, tests generated backchannels and general full-duplex abilities on a benchmark, and uses listener ratings to assess appropriateness.
4.1. Probing
- Probing compares base and backchannel-head-trained models by fitting layerwise onset predictors on TurnBench and testing on held-out conversations against human annotations.The probe scores both predicted frequency and whether onset timing matches the annotated timing.Source: §4.1 · p. 3
4.2. Evaluating the Generated Backchannels
- Full-Duplex-Bench evaluation compares augmented models with base models and RL-based shaping across backchannel metrics, pause handling, turn-taking, and interruption.Three thresholds are tested: normal matches the human TurnBench rate, while low and max yield fewer and more backchannels.Source: §4.2 · p. 3
4.3. Human Evaluation
- Human evaluation rates human, random-baseline, and model versions of 35 short TurnBench samples for backchannel appropriateness.Four annotators rate each version on a 1–5 scale.Source: §4.3 · p. 3
5. RESULTS
The section evaluates backchannel frequency, onset timing, generated forms, and human judgments. The results include improved benchmark timing over base models, with trade-offs against RL-based shaping, and human ratings comparable to real backchannels.
5.1. Probing
- PersonaPlex-BC reaches the human backchannel rate when probing its last layer at a threshold around 0.5; F-Actor-BC improves over its base model by less.The frequency shift suggests the mechanism moves hidden states toward human-like backchannel frequency.Source: §5.1 · p. 4
- PersonaPlex-BC has substantially higher onset-timing F1 than base PersonaPlex across nearly all layers, while F-Actor-BC’s middle-layer gain narrows to a small last-layer improvement.Both models score better with a two-frame (±0.16s) tolerance around annotated onset; F-Actor(-BC) performs best at low decision thresholds.Source: §5.1 · p. 4
5.2. Evaluating the Generated Backchannels
- BC variants backchannel more often and with lower JSD than base models; PersonaPlex-BCmax beats RL on frequency, JSD, and turn-taking speed but has higher take-over rates, especially on pauses.F-Actor’s pause TOR at the normal setting improves over its base model and is among the best in the table.Source: §5.2 · p. 4, Table 1 · p. 4
- Free and training-token-path-constrained generation yield similar lexical distributions, including roughly double the human share of “yeah,” but constrained audio sounds more natural.The naturalness comparison is based on listening to the generated audio.Source: §5.2 · p. 4
5.3. Human Evaluation
- PersonaPlex-BC backchannels score 4.37/5 for appropriateness versus 4.34/5 for human recordings and 3.77/5 for random placement; both human and model ratings exceed random, with no significant difference between them.The human-versus-model comparison has p = 0.47; inter-annotator agreement is moderate (Krippendorff’s α = 0.42).Source: §5.3 · p. 4, Figure 3 · p. 4
6. DISCUSSION AND CONCLUSION
The conclusion presents backchannel timing as a lightweight, pluggable signal rather than an implicit behavior learned from conversational training. Across two architectures, the method improves backchannel frequency and timing, and its backchannels are judged on par with human ones.
6. DISCUSSION AND CONCLUSION
- Across two architectures, explicit backchannel timing control improves frequency and timing, and human evaluators judge the generated backchannels on par with human ones.The authors release the models and code to support further work on the control mechanism.Source: §6 · p. 4