PaperlayerView exact revision on arXiv

Source-linked AI summary

Why Full-Duplex Speech Models Collapse in Long Conversations

Yeonju Kim, Bella Godiva, Se Jin Park, Yong Man Ro

arXiv:2610.32349v1cs.CLcs.SD

TL;DR

Full-duplex models generate continuously while listening, but existing benchmarks do not establish whether they remain stable over long conversations. FD-LHB tests five models in replayed and adaptive conversations, finding behavioral degradation or context exhaustion; two training-free interventions reduce collapse in replays of up to an hour.

  • Problem

    Existing benchmarks cover single turns or a few rounds, leaving long-horizon full-duplex stability and failure modes systematically uninvestigated.

    Source: §1 · p. 1

  • Method

    FD-LHB evaluates five models through controlled conversation replay and adaptive interaction, then tests cache pinning and repetition-penalty interventions.

    Source: §1 · p. 2

  • Results

    Across five models, sustained operation is limited by behavioral degradation or context exhaustion; combined IAP and DRP reduce repetition, silence, and gibberish in replays of up to an hour.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The intervention results support the diagnosis, while the authors do not claim that these fixes solve long-horizon full-duplex modeling.

    Source: §1 · p. 2, §7 · p. 9

    The training setup is confirmed in Moshi’s official fine-tuning code, but whether pretraining and PersonaPlex training use it is not public.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

Full-duplex speech models listen and speak at the same time, which makes them promising for natural, open-ended conversation. But can they sustain a coherent conversation for tens of minutes? Because they generate a token every frame, even while listening, they exhaust their context windows within minutes. Yet existing benchmarks evaluate only single turns or a few conversational rounds, too short to expose the long-horizon problem. We introduce the Full-Duplex Long-Horizon Benchmark (FD-LHB), which tests five recent full-duplex models on long replayed conversations and in live conversation with an LLM partner. Every evaluated model eventually either degrades or exhausts its context. Models that keep the full history often degrade before their context runs out, while sliding-window models still fall into repetition, silence, and gibberish. For sliding-window models, we identify two contributing mechanisms. Gibberish and silence are linked to Initialization-Anchor Eviction, where the sliding window discards the first KV entry. Repetition is also reinforced by the model's own recent outputs. Guided by this diagnosis, we apply two simple fixes: Initialization-Anchor Pinning (IAP) keeps that first entry, and Duplex Repetition Penalty (DRP) discourages echoing while guarding against prolonged silence. Together, they support the diagnosis and, without retraining, substantially reduce conversational collapse in replays of up to an hour while running in real time.

1 INTRODUCTION

Full-duplex models can lose coherence during long conversations because they generate tokens continuously, while existing benchmarks have not systematically tested sustained interaction. FD-LHB evaluates long conversations, diagnoses collapse mechanisms, and tests training-free fixes.

1 INTRODUCTION

  • Continuous frame-by-frame generation, including while listening, fills context with silence and recurring backchannels, and generated output can influence later generation.Existing benchmarks evaluate single turns or a few rounds rather than many minutes of interaction.Source: §1 · pp. 1–2

Systematic characterization and diagnosis of collapse

  • Across five recent models, long conversations end in behavioral degradation or context exhaustion, including sliding-window collapse and growing-context collapse before the limit.Observed failures include repetition, prolonged silence, and gibberish.Source: §1 · p. 2
  • Silence and gibberish are linked to Initialization-Anchor Eviction, while repetition is reinforced by the model’s own recent outputs.The first mechanism concerns eviction of the first KV entry computed without prior cached context.Source: §1 · p. 2

Validation through training-free interventions

  • Initialization-Anchor Pinning keeps the first KV entry, while Duplex Repetition Penalty discourages recently used words without making the model less likely to speak.Together, the fixes substantially reduce repetition, silence, and gibberish in both sliding-window models; the authors do not claim to solve long-horizon modeling.Source: §1 · p. 2

FD-LHB benchmark

  • FD-LHB tests long-horizon interaction with 40 controlled replays up to 60 minutes and 40 adaptive sessions up to 20 minutes.The benchmark combines replayed human conversations with an LLM-based spoken partner.Source: §1 · p. 2

Validation through training-free interventions

  • Gibberish falls from 100% to 0%; repetition falls from 65% to 0% in Moshi and 90% to 2.5% in PersonaPlex; prolonged silence falls from 100% to 7.5% and 85% to 47.5%, respectively.The interventions require no retraining.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work evaluates full-duplex interaction but not stability over tens of minutes. This paper connects long-horizon inference to cache management and output feedback, examining the first KV entry and repetition control.

2.1 FULL-DUPLEX SPEECH GENERATION AND STREAMING INTERACTION

  • Earlier full-duplex benchmarks test turn-taking, overlap, interruption handling, multi-round quality, and task completion, but not stability over extended conversations.The paper evaluates repetition, silence, and gibberish during continuous generation lasting tens of minutes.Source: §2.1 · p. 3

2.2 LONG-HORIZON STABILITY AND KV-CACHE MANAGEMENT

  • Training-based methods address training–generation mismatch, while this work studies continual-inference instability and uses DRP to modify token selection without retraining.The paper frames generated outputs reinforcing later repetition as the inference-time feedback it targets.Source: §2.2 · p. 3
  • Prior cache strategies retain initial attention sinks or select tokens by attention; IAP preserves only the first KV entry and updates its position index.In Moshi and PersonaPlex, the first KV entry alone prevents gibberish in all tested runs, unlike the tested later entries; IAP avoids re-indexing the whole cache as StreamingLLM does.Source: §2.2 · p. 3

3 FULL-DUPLEX LONG-HORIZON BENCHMARK

FD-LHB combines controlled replay with adaptive conversation to evaluate long-horizon behavior. It tracks the onset of repetition, prolonged silence, and gibberish using specified window and bin criteria.

3.1 EVALUATION PROTOCOL

  • FD-LHB pairs fixed-input CANDOR replay with adaptive LLM-partner sessions, balancing controlled comparison against interactive responses that differ across runs.The two tracks are complementary because each addresses the other’s limitation.Source: §3.1 · p. 3
  • CANDOR replay selects the 40 longest conversations after removing their first three turns; durations range from 55.5 to 92.6 minutes, with a 67.4-minute mean.Each model receives the same fixed speaker-channel input and is evaluated for up to 60 minutes or until the conversation ends.Source: §3.1 · pp. 3–4
  • Adaptive sessions use Gemini-2.5-Flash to generate partner turns, synthesized by a local VITS model, with 40 sessions per model lasting up to 20 minutes.Sessions begin from one of ten everyday topics, and the partner responds to the evaluated model’s output.Source: §3.1 · p. 4

3.2 FAILURE MODES AND ONSET METRICS

  • The benchmark tracks repetition, silence, and gibberish, defining onset as the first window or bin that meets the corresponding criterion.Repetition uses 32-word windows; silence and gibberish use one-minute bins.Source: §3.2 · p. 4
  • Repetition onset is the first of two consecutive 32-word windows with Seq-Rep-4 ≥0.30, so a single repeated phrase does not count.Seq-Rep-4 captures repeated four-word patterns in recent output.Source: §3.2 · p. 4
  • Prolonged-silence onset requires two consecutive bins with fewer than three spoken words; gibberish onset is the first eligible one-minute bin judged gibberish by Gemini.Gibberish bins must contain at least three words; the judge sometimes misses real gibberish, so reported rates may be slightly low.Source: §3.2 · p. 4

4 DIAGNOSING LONG-CONVERSATION COLLAPSE

The section investigates why long-conversation collapse occurs, finding cache-turnover links for silence and gibberish and self-reinforcement for repetition. Tests show that retaining the first cache entry prevents gibberish in all tested runs.

4 DIAGNOSING LONG-CONVERSATION COLLAPSE

  • The section frames its analysis around the three observed collapse modes: repetition, silence, and gibberish.The mechanisms are investigated after these failure modes are observed in long-horizon evaluation.Source: §4 · p. 4

4.1 INITIALIZATION-ANCHOR EVICTION

  • Training starts each chunk with an initialization anchor, but continual inference eventually evicts that first KV entry, creating a training–inference mismatch.The paper hypothesizes this eviction contributes to collapse, and tests the hypothesis through cache-window sweeps and entry-retention experiments.Source: §4.1 · pp. 4–5, Figure 3 · p. 5

4.1.1 COLLAPSE TIMING FOLLOWS CACHE TURNOVER

  • Silence and gibberish onset scale roughly with attention-window length W, while repetition shows no consistent relation to W.With conversational input, onset is (1.63±0.49)W for silence and (2.70±0.20)W for gibberish; with silence-only input, gibberish onset is (2.54±0.19)W.Source: §4.1.1 · p. 5, Figure 4 · p. 5

4.1.2 THE FIRST KV ENTRY IS SUFFICIENT

  • Keeping only the first cache entry prevents gibberish in every tested run; keeping more initial entries adds no improvement.Every alternative single early entry failed in at least one run, while all tested settings with P≥1 had overlapping gibberish-free curves.Source: §4.1.2 · p. 5, Figure 5 · p. 6

4.2 RECENT OUTPUTS REINFORCE REPETITION

  • A word’s share rises before repetition onset, and increasing forced insertions of purple increases its recurrence over the next 20 seconds.The increase appears in both models and is strongest in PersonaPlex.Source: §4.2 · p. 6, Figure 6 · p. 6

5 TESTING THE DIAGNOSIS WITH TRAINING-FREE INTERVENTIONS

The paper tests its collapse diagnosis with two training-free inference interventions: IAP retains the initialization anchor, while DRP reduces repetition and guards against prolonged silence.

5 TESTING THE DIAGNOSIS WITH TRAINING-FREE INTERVENTIONS

  • The two training-free interventions test whether retaining the initialization anchor and limiting recent-word reinforcement reduce their associated failures.Both operate at inference time and can be applied to existing models.Source: §5 · p. 6

5.1 INITIALIZATION-ANCHOR PINNING (IAP)

  • IAP preserves the first KV entry while otherwise sliding the cache as usual.When the cache is full, it removes the oldest entry after the anchor; the anchor’s position index increases as the window slides.Source: §5.1 · pp. 6–7

5.2 DUPLEX REPETITION PENALTY (DRP)

  • DRP penalizes recently used assistant text words while leaving audio streams unchanged, and adjusts PAD during prolonged silence to reduce silent periods.It penalizes words appearing in the last 512 lexical tokens and gradually lowers the PAD logit when the model has been silent and the user is quiet.Source: §5.2 · p. 7

6 RESULTS AND ANALYSIS

Across five models, baselines either degrade or reach context capacity; IAP+DRP reduces collapse in the sliding-window models and preserves response quality. Ablations distinguish the components’ contributions, and IAP runs faster than a StreamingLLM-style baseline.

6.1 EXPERIMENTAL SETUP

  • The five models use either sliding-window or growing-context management; growing-context runs end at capacity, while IAP+DRP is applied to the two sliding-window models.The growing-context models reach capacity after about 5, 17.5, and 10.9 minutes, respectively; capacity is treated as a run end, not a collapse event.Source: §6.1 · p. 7, §6.2 · p. 7

6.2 MAIN RESULTS

  • IAP+DRP reduces repetition, silence, and gibberish across replay and adaptive interaction while keeping coherence and context scores steady.On CANDOR replays, failure rates fall to 0%, 7.5%, and 0% for Moshi and 2.5%, 47.5%, and 0% for PersonaPlex; in adaptive sessions, each failure occurs in no more than 10% of sessions for both models.Source: Figure 7 · p. 8, §6.2 · p. 8, Figure 8 · p. 8

6.3 ABLATION

  • In the Moshi replay ablation, IAP addresses gibberish and most silence, DRP addresses repetition, and only their combination suppresses all three failures.IAP raises the silence-free share from 0% to 67.5%, and adding DRP raises it to 92.5%; IAP alone keeps every run free of gibberish.Source: §6.3 · p. 9, Figure 9 · p. 9

6.4 REAL-TIME INFERENCE COST

  • Both IAP and the StreamingLLM-style baseline prevent gibberish in every run, but IAP’s RTF is 0.75 versus 1.03 for the baseline.The paper attributes IAP’s faster processing to updating only the anchor rather than re-indexing all remaining entries.Source: Table 1 · p. 9, §6.4 · p. 9

7 CONCLUSION

The conclusion presents FD-LHB as an evaluation of long conversations using controlled replay and adaptive interaction. Across five models, operation is limited by degradation or context exhaustion; IAP+DRP reduces collapse without retraining and runs in real time.

7 CONCLUSION

  • FD-LHB evaluates whether full-duplex speech models sustain long conversations through controlled replay and adaptive interaction.The conclusion reports that five evaluated models encounter behavioral degradation or context exhaustion.Source: §7 · p. 9
  • IAP retains the initialization anchor, while DRP discourages repetition and guards against prolonged silence; together they reduce repetition, silence, and gibberish.The combined interventions are reported to improve stability in adaptive conversations with an LLM partner as well as in replays up to an hour.Source: §7 · p. 9
  • The combined methods require neither retraining nor additional memory and run in real time.Source: §7 · p. 9

ETHICS STATEMENT

The appendix documents FD-LHB’s evaluation inputs, metrics, validation, and reproducibility, alongside participant and corpus protections.

ETHICS STATEMENT

  • Listening studies used consenting adult, English-proficient Prolific participants; CANDOR audio is accessed under research terms and is not released.The release includes a manifest of conversation IDs and crop boundaries, not the audio.Source: ETHICS STATEMENT · p. 10

A.1 BENCHMARK INPUTS AND MODEL SETTINGS

  • FD-LHB combines controlled CANDOR replays with adaptive LLM-partner interaction, using identical 16-kHz mono inputs across model conditions.The replay set uses the 40 longest conversations after removing the first three turns; adaptive sessions last up to 20 minutes or the model’s context limit.Source: §A.1 · p. 14

A.2 METRICS, JUDGES, AND AGGREGATION

  • Repetition and prolonged silence require two consecutive threshold-crossing windows or bins, while gibberish is judged per speaking bin and quality scores cover Coherence and Context.Runs ending at a context limit or input end are right-censored in time-to-event curves.Source: §A.2 · pp. 15–16

A.3 HUMAN VALIDATION OF THE GEMINI GIBBERISH JUDGE

  • Gemini’s gibberish labels matched the human majority on 36 of 40 clips (Cohen’s κ = 0.80), but humans identified four clips Gemini missed.The authors describe the label as a reasonable proxy, while noting audio-versus-transcript judging and an artificially balanced clip sample.Source: §A.3 · p. 16

A.4 CHECKING THE COLLAPSE METRICS ON AUDIO

  • Text-based silence labels closely matched audio-based labels (AUROC 0.983), while text-based repetition had high precision but missed some audible repetition.Text repetition precision was 1.00 with Whisper small.en and 0.98 with large-v3; recall was 0.72 and 0.68, respectively.Source: §A.4 · pp. 16–17, Table 6 · p. 17

A.5 WHY PREDICTIVE ENTROPY INSTEAD OF PERPLEXITY

  • Across 2,750 speaking bins, PPL ranges overlapped between gibberish and non-gibberish, whereas predictive entropy separated the groups; the authors use entropy only for mechanism analyses.Predictive entropy requires model logits, which not every evaluated model exposes.Source: §A.5 · p. 17

A.6 SENSITIVITY TO METRIC THRESHOLDS

Sensitivity checks show that the reported repetition and gibberish comparisons are robust to tested thresholds, while prompt and release details specify the evaluation procedure.

A.6 SENSITIVITY TO METRIC THRESHOLDS

  • Across every tested repetition setting, baselines were flagged in at least 9 of 20 runs, while IAP+DRP was flagged in none under the two-window rule and at most 3 under the single-window rule.The checks vary Seq-Rep-4 thresholds, window sizes, and the persistence rule.Source: §A.6 · pp. 17–18, Table 8 · p. 19
  • A stricter three-consecutive-bin gibberish rule leaves replay outcomes unchanged for the sliding-window models, but hides most gibberish in BayLing-Duplex and Lychee-FD.The authors therefore retain the single-bin rule.Source: §A.6 · p. 18

A.7 GEMINI PROMPTS

  • Gemini prompts distinguish gibberish from understandable repetition or irrelevant speech, and score Coherence and Context on separate 1–5 scales.The interactive partner receives recent dialogue and responds in short turns; its prompt uses everyday topics.Source: §A.7 · p. 19

A.8 REPRODUCIBILITY

  • The planned release includes code, partner and judge prompts, metric computation, and a CANDOR manifest, but not the research-access audio.The appendix also specifies diagnostic analyses of initialization-anchor eviction and self-reinforcing repetition.Source: §A.8 · p. 20, B DIAGNOSTIC EXPERIMENTS: DETAILS AND… · p. 20

B.1 INITIALIZATION-ANCHOR EVICTION

The diagnostic experiments link sliding-window failure to eviction of the initialization anchor and show that the pinned entry serves as an attention sink.

B.1 INITIALIZATION-ANCHOR EVICTION

  • Keeping only the first cache entry prevented high late-conversation entropy in every run; keeping no entry produced high entropy, while retaining any number of first entries from one onward kept it low.The identity test found that no other single early entry prevented gibberish in all runs.Source: §B.1 · p. 20, Figure 11 · p. 21
  • Shifting or re-indexing position indices did not change the high-entropy onset, which remained 12.1 ± 1.1 minutes.This result argues against large position indices as the cause in the tested settings.Source: §B.1 · p. 21, Figure 12 · p. 21
  • With IAP, the pinned anchor received about 32% of attention during the first window and more than 60% after the first four-minute turnover.The measurement comes from one 14-minute PersonaPlex replay.Source: Figure 13 · p. 21, §B.1 · p. 22

B.2 SELF-REINFORCING REPETITION

  • In PersonaPlex, forced occurrences of a previously absent word increased its subsequent share to 28.4% at 32 insertions; Moshi stayed below 10% at every tested dose.Without insertion, neither model said the word during the free-generation period.Source: §B.2 · p. 22

C IMPLEMENTATION DETAILS OF IAP AND DRP

IAP preserves a cache anchor and DRP reduces repetition while guarding against prolonged silence; together, they improve long-replay behavior without retraining.

C.1 INITIALIZATION-ANCHOR PINNING

  • IAP pins the first KV entry while the remaining L−1 cache slots continue normal FIFO updates, keeping cache size fixed without retraining.The pinned key is repositioned as the window slides so its relative position remains controlled.Source: §C.1 · p. 22

C.2 DUPLEX REPETITION PENALTY

  • DRP penalizes recently used lexical tokens while preserving total lexical probability, then lowers the PAD logit after prolonged silence when the user is quiet.This design aims to discourage repetition without making the model less likely to speak.Source: §C.2 · pp. 22–23

C.3 FINAL SETTINGS

  • The final configuration uses a 512-token history, a single pinned anchor, and model-specific repetition-penalty strengths of 1.5 for Moshi and 1.3 for PersonaPlex.The PAD correction begins after 30 seconds of silence and ramps to its maximum over 10 seconds.Source: §C.3 · p. 23

D EXTENDED RESULTS

  • The appendix extends the main results with full benchmark statistics, response-quality trajectories, silence duration, component ablations, hyperparameter tests, and computational cost.Table 10 reports onset times over 40 runs per condition, with context limits and input endings treated as right-censored.Source: D EXTENDED RESULTS · p. 23, Table 10 · p. 24

D.1 COMPLETE MAIN RESULTS

  • Across six matched comparisons, IAP+DRP avoided each failure in more replays than baseline, with all six exact McNemar tests significant after Bonferroni correction.Each comparison covers two models and three failure types.Source: §D.1 · p. 24, Table 10 · p. 24

D.2 LONG-HORIZON TRAJECTORIES AND RESPONSE QUALITY

  • With IAP+DRP, repetition and gibberish stayed near zero through 60-minute replays, words per minute stayed steady, and both quality scores remained stable for the full session.Without the combined method, the sliding-window baselines’ Coherence and Context scores fell to their lowest values within about 20 minutes.Source: §D.2 · p. 24, Figure 14 · p. 25, Figure 15 · p. 25

D.3 SILENCE DURATION AND RESUMPTION OF SPEECH

  • Two-minute silences fell from 40 to 3 runs for Moshi and from 34 to 19 for PersonaPlex with IAP+DRP.Most remaining PersonaPlex silences were temporary: 18 of 19 runs resumed speaking.Source: §D.3 · p. 26, Table 11 · p. 26

D.4 COMPONENT ABLATIONS

  • On 40 matched Moshi replays, IAP+DRP gave the best result on all three failures; IAP alone removed gibberish, while DRP alone removed repetition.The component results support complementary roles but do not establish that each failure has a single cause.Source: §D.4 · p. 26

D.5 DRP ABLATION

The appendix tests DRP components and alternatives, checks sensitivity and listener preferences, and measures inference cost. Additional-model observations are qualitative because protocols differ and each model has one sequence.

D.5 DRP ABLATION

  • On 40 PersonaPlex replays, full DRP reduced repetition to 0 runs and silence to 5, versus 34 and 4 with IAP alone.Standard repetition penalty reduced repetition to 5 runs but raised silence to 21; mass-preserving penalty reduced those counts to 2 and 13 before the PAD safeguard lowered silence to 5 and repetition to zero.Source: §D.5 · pp. 26–27, Table 13 · p. 27

D.6 ALTERNATIVE REPETITION CONTROLS

  • On PersonaPlex, DRP and Bayesian both prevented thresholded repetition, while DRP had 5 silent runs versus 7 for Bayesian.LZ reached repetition in 31 of 40 runs, and standard repetition penalty produced silence in 21; DRP needs only recent lexical tokens, without LZ's compressor state or Bayesian's separately estimated prior.Source: Table 14 · p. 28, §D.6 · p. 28

D.7 HYPERPARAMETER SENSITIVITY

  • ρ=2.0 left 1 silent run but drove output to 244.2 words/min, so fewer thresholded failures did not identify a uniquely best setting.Changing history length from 256 to 1024 tokens changed repetition by at most one run; the authors treat the final setting as a fixed operating point, not a uniquely optimal choice.Source: §D.7 · p. 28, Table 15 · p. 29

D.8 HUMAN LISTENING COMPARISON

  • Listener preferences were near neutral overall (0.47–0.53), favoring IAP alone early and IAP+DRP mid-conversation; late intervals included 0.5.Early preferences favored IAP alone for intelligibility and context, while mid-conversation preferences favored IAP+DRP for naturalness; late-phase intervals showed no clear preference.Source: Table 16 · p. 29, §D.8 · p. 29

D.9 COMPUTATIONAL COST

  • IAP+DRP ran in real time at RTF 0.75, while StreamingLLM-style overhead was +73.3% and RTF exceeded 1.DRP added 2.7% cost and IAP added 24.0%; StreamingLLM-style re-indexing of the full cache was the slowest method.Source: Table 17 · p. 30, §D.9 · p. 30

E EXPLORATORY OBSERVATIONS AND LIMITATIONS

  • Exploratory additional-model results are not pooled with FD-LHB because the protocols differ.Each model uses its own prompt, input, context unit, and stopping rule, so the appendix treats these observations as qualitative.Source: E EXPLORATORY OBSERVATIONS AND… · p. 30

E.1 EXPLORATORY OBSERVATIONS ON ADDITIONAL SPEECH MODELS

  • Seven of nine additional models repeated in one tested sequence; GLM-4-Voice and LLaMA-Omni2 stayed stable, but protocols preclude ranking.Only one sequence per model was observed, and lengths and protocols differed.Source: Table 18 · p. 30, §E.1 · p. 30

E.2 LIMITATIONS

The limitations concern the diagnosis's scope, finite observation windows and cross-model comparability, and evaluation metrics that do not fully capture conversation quality.

E.2 LIMITATIONS

  • The diagnosis covers only Moshi and PersonaPlex; it does not explain growing-context models' collapse, and IAP does not apply to models that never evict cache entries.The fixed-slot test measures predictive entropy rather than dialogue quality, and broader tests across models, inputs, seeds, architectures, and languages are needed.Source: §E.2 · p. 31
  • A finite run with no observed failure does not establish indefinite stability, and comparisons within a model are more interpretable than cross-model comparisons.Evaluated models differ in architecture, context management, and generation settings.Source: §E.2 · p. 31
  • Avoiding repetition, silence, and gibberish does not guarantee coherent conversation; PersonaPlex still had two-minute silence in 19 of 40 IAP+DRP replays.The gibberish detector may miss cases, and listeners favored IAP alone early in the conversation, indicating possible trade-offs from DRP.Source: §E.2 · p. 31

Related papers

Loading 2610.32349v1…