PaperlayerView exact revision on arXiv

Source-linked AI summary

Diagnosing and Mitigating Long-Session Degradation in Full-Duplex Speech Models

Junho Kim, CheHyun Lee, Donghun Kang, Youngho Cho, Dongwook Lee, Heeseung Kim, Sungroh Yoon

arXiv:2610.34991v1cs.CL

TL;DR

Long-session behavior in full-duplex spoken dialogue models remains insufficiently understood, despite evaluations of their interaction capabilities. This paper uses closed-loop evaluation and proposes self-distillation and two-resolution context management. Across three models, the methods improve stability, with TRC extending gains beyond the training horizon and reducing KV-cache growth by more than 90%.

  • Problem

    How full-duplex spoken dialogue models’ behavior evolves over extended continuous sessions remains insufficiently understood, despite benchmarks of diverse interaction capabilities.

    Source: §1 · p. 2

  • Method

    The paper combines closed-loop evaluation of long-session behavior with privileged-context self-distillation and a two-resolution context architecture that preserves recent interactions and compresses earlier history.

    Source: §1 · p. 2

  • Results

    Across three models, self-distillation improved stability within the training horizon, while adding TRC better preserved spoken content and voice consistency beyond it and reduced KV-cache growth by 91.7–95.8%.

    Source: §1 · p. 2, §6 · p. 10

  • Takeaways & Limitations

    TRC sustained coherent responses and voice consistency in 20-minute conversations, extending improvements beyond the training horizon where self-distillation alone showed severe behavioral collapse.

    Source: §1 · p. 2

    The simulator did not initiate interruptions or backchannels, so long-session stability under those user behaviors was untested.

    Source: D LIMITATIONS AND FUTURE WORK · p. 36

Abstract

from arXiv · show

End-to-end full-duplex spoken dialogue models enable natural, real-time spoken interactions with interruptions, backchannels, and overlapping speech. Prior evaluations have assessed diverse interaction capabilities and identified some degradation patterns across conversational turns. However, how model behavior evolves over extended sessions remains insufficiently understood. In this work, we address this gap by systematically characterizing long-session degradation and developing efficient methods to mitigate it. We introduce a closed-loop evaluation framework for long-session stability and apply it to several publicly available full-duplex speech models, revealing degradation over time in responsiveness, spoken content quality, and voice consistency. To mitigate this degradation, we propose a self-distillation-based training framework for lightweight LoRA adaptation that improves long-session stability with a small amount of additional training. We further introduce a two-resolution context (TRC) architecture that preserves recent interactions in full detail while compressing older interaction history into key-value representations. We evaluate our methods on three models with distinct architectures, assessing long-session stability, information retention, and computational efficiency. Across all three models, we show that LoRA adaptation mitigates performance degradation over ten-minute sessions, while adding TRC extends these gains beyond the training session length and reduces KV-cache growth by more than 90%.

1 INTRODUCTION

Full-duplex models support more natural, continuous spoken interaction, but their behavior over extended sessions remains insufficiently understood. This work characterizes long-session degradation and proposes training and context-management methods to mitigate it.

1 INTRODUCTION

  • End-to-end full-duplex research increasingly targets sophisticated real-time interaction, including tool use and multimodal perception, making reliable extended interaction an open need.The section frames sustained interaction as necessary for interactive assistants that continuously perceive their surroundings and respond as situations unfold.Source: §1 · p. 1
  • Although benchmarks assess diverse full-duplex capabilities, the evolution of behavior and performance during extended continuous sessions remains insufficiently understood.The authors state that, to their knowledge, prior work had not explicitly targeted long-session stability in end-to-end full-duplex models through dedicated training or architectural designs.Source: §1 · p. 2
  • The proposed diagnostic framework uses closed-loop simulations to track responsiveness, spoken-content quality, and voice consistency as conversations progress.The simulations revealed diminished responsiveness, semantically incoherent utterances that could become non-speech vocalizations, and vocal-characteristic drift.Source: §1 · p. 2
  • Privileged-context self-distillation links short sessions into long conversational trajectories to create training targets without an external full-duplex dialogue corpus.TRC preserves recent interactions at full resolution and compresses earlier context into compact KV-cache representations.Source: §1 · p. 2
  • Across three distinct model architectures, combining self-distillation with TRC reduced spoken-content and voice-consistency degradation by 71–96%, sustained stability in 20-minute conversations, and reduced KV-cache growth by 91.7–95.8%.Without TRC, self-distillation alone showed severe behavioral collapse beyond the training horizon; TRC also limited inference-latency increases to 12–15% in models with growing context windows, versus 141–259% without TRC.Source: §1 · p. 2

2 RELATED WORK

Related work covers full-duplex interaction modeling, evaluations of dialogue capabilities and performance over time, and streaming approaches to long-context management. This paper extends those lines toward sustained spoken interaction.

2.1 FULL-DUPLEX SPEECH MODELING

  • Full-duplex models can integrate turn-taking into generation, modeling speech content and conversational timing together (Nguyen et al., 2023; D´efossez et al., 2024).Other approaches use dedicated turn-taking mechanisms, including external controllers and predictors based on dialogue-model representations.Source: §2.1 · p. 2
  • Full-duplex speech and text histories accumulate linguistic, acoustic, and interaction-related information that may become difficult to process reliably as conversations unfold.The resulting difficulties may affect recall and response content as well as speech quality and basic responsiveness, including whether and when a model responds.Source: §2.1 · p. 3

2.2 EVALUATION OF FULL-DUPLEX SPOKEN DIALOGUE MODELS

  • Prior benchmarks assess interaction dynamics, multi-turn coherence and instruction following, and tool use, but analyses of performance over time have focused on conversational rounds or elapsed time.This work extends that analysis to longer, dynamically evolving conversations and examines responsiveness, spoken-content quality, and voice consistency.Source: §2.2 · p. 3

2.3 STREAMING LONG-CONTEXT MODELING

  • Streaming context methods use local raw-token windows with compressed representations of older context, but have primarily been studied in text-based language modeling.This work adapts incremental context compression to full-duplex models that jointly generate speech and conversational timing.Source: §2.3 · p. 3

3 DIAGNOSING LONG-SESSION DEGRADATION

The authors evaluate long-session behavior through closed-loop simulated conversations and track responsiveness, spoken-content quality, and voice consistency. The four baseline models degrade in different ways and at different points during sessions.

3.1 PROTOCOL AND MEASURES

  • The evaluation simulated 90 English conversations per model, each targeting 600 seconds, with subsequent user turns conditioned on the model’s generated speech.The four evaluated models were Moshi, PersonaPlex, MiniCPM-o 4.5, and Raon-SpeechChat.Source: §3.1 · p. 3
  • The simulator waits for the model to finish before replying and does not generate backchannels or interruptions, leaving stability under those user behaviors untested.Logical time follows processed-audio duration rather than wall-clock time, so inference and processing latency do not affect timing measurements.Source: §3.1 · p. 4
  • Six metrics track three aspects of interaction in one-minute logical-time intervals: responsiveness, spoken-content quality, and voice consistency.The measures include FTO, response timeout and terminated-session rates, conversation quality, WER, and speaker similarity.Source: §3.1 · p. 4

3.2 LONG-SESSION DEGRADATION PATTERNS

  • PersonaPlex and Raon-SpeechChat conversation quality fell from 4.90 to 1.08 and from 4.62 to 1.03, respectively, while final-minute WER reached 97.9% and 98.7%.PersonaPlex degraded progressively, while Raon-SpeechChat’s degradation became prominent in the final two to three minutes.Source: §3.2 · p. 5
  • The four baseline models showed distinct degradation patterns; MiniCPM-o 4.5’s median FTO increased from 0.39 s in the first minute to 1.89 s at 8–9 min, and its final-minute response timeout rate reached 75.8%.MiniCPM-o 4.5’s mean conversation quality remained above 4 among sessions with valid observations, while WER rose from 2.4% to 15.5%. Moshi’s response timeout rate rose from 12.7% at 7–8 min to 70.2% at 8–9 min, and its cumulative terminated-session rate reached 96.7% by 10 min.Source: §3.2 · pp. 4–5

4 METHODS FOR LONG-SESSION STABILITY

The methods combine privileged-context self-distillation, which transfers short-session behavior to longer trajectories, with TRC, which keeps recent dialogue detailed and compresses older history. Together they target long-session stability without requiring an external full-duplex dialogue corpus.

4.1 PRIVILEGED-CONTEXT SELF-DISTILLATION

  • Privileged-context self-distillation trains a student on concatenated dialogue trajectories using predictions from the frozen base model as teacher targets.The teacher generates short segments with preceding-dialogue summaries in its prompt, then quality-filters them before concatenation.Source: §4.1 · pp. 5–6
  • The student processes the trajectory continuously under teacher forcing, while the teacher conditions each segment on its summary and within-segment context.At aligned positions, the student has no access to the teacher’s summaries.Source: §4.1 · p. 6, Figure 3 · p. 6
  • Training separately matches teacher-to-student content and control predictions, then matches hidden representations supplied to the frozen audio decoder.Separating content and control KL losses allows their relative weights to be set independently.Source: §4.1 · p. 6

4.2 TWO-RESOLUTION CONTEXT (TRC) ARCHITECTURE

  • TRC combines recent raw dialogue with compact gist representations of older tiles, retaining a bounded local window while preserving access to earlier context.Each tile gets a learned gist token; raw tokens are discarded once their tile leaves the local window.Source: §4.2 · pp. 6–7
  • TRC gist tokens incorporate their tile’s raw tokens and all preceding gists, allowing earlier information to propagate after raw tokens are discarded.The model trains the shared gist embedding and LoRA adapters on backbone Q/K/V/O projections while keeping audio modules frozen.Source: §4.2 · p. 7
  • TRC is trained with privileged-context self-distillation using the same attention visibility and positional rules as streaming inference.At inference, gist memory grows by one token per tile while raw context remains bounded by the local window.Source: §4.1 · p. 6, §4.2 · p. 7

5 EXPERIMENTS

The experiments assess self-distillation and TRC on three full-duplex models, comparing long-session stability, memory recall, and computational costs. Self-distillation generally helps within its training horizon, while TRC better retains spoken-content and voice-consistency gains beyond it.

5.1 LONG-SESSION STABILITY

  • Within the ten-minute training horizon, self-distillation generally improves stability; PersonaPlex and Raon-SpeechChat reach 8–10-minute conversation-quality scores of 3.84 and 4.19 with TRC.For MiniCPM-o 4.5, responsiveness degradation remains substantial and TRC adds no within-horizon benefit over LoRA alone.Source: §5.1 · pp. 7–8, Table 1 · p. 8
  • At 18–20 minutes, TRC retains more spoken-content and voice-consistency gains than LoRA alone, which deteriorates sharply beyond the training horizon.Across the three models, LoRA-only WERs range from 88.7% to 99.6%; TRC results vary by model, including PersonaPlex conversation quality of 3.53 and MiniCPM-o 4.5 WER of 5.8%.Source: §5.1 · p. 8, Table 1 · p. 8

5.2 COMPUTATIONAL EFFICIENCY

  • 91.7–95.8%: TRC reduces prompt-excluded KV growth rate, with latency trade-offs varying by model and context length.Latency measurements cover the backbone only; PersonaPlex remains slower at measured points, while reduced attention cost increasingly offsets gist-computation overhead for MiniCPM-o 4.5 and Raon-SpeechChat.Source: §5.2 · p. 9, Table 2 · p. 9

5.3 FURTHER ANALYSIS

  • TRC generally sacrifices some short-term recall relative to Base but sustains recall as sessions lengthen, an advantage where Base recall degrades or is lost.The authors identify the absence of explicit memory tasks in training as a limitation and note a compression-level trade-off.Source: §5.3 · p. 9, Figure 4 · p. 10
  • 12.8% to 19.8%: removing hidden-state supervision raises late-session WER, while speaker similarity falls from 0.579 to 0.418.The PersonaPlex ablation keeps other parameters fixed.Source: §5.3 · p. 9, Table 3 · p. 9

6 CONCLUSION

The paper documents long-session degradation and introduces self-distillation and TRC to mitigate it; its appendices specify the evaluation setup and user-simulator instructions.

6 CONCLUSION

  • All four evaluated models degraded in at least one dimension; self-distillation improved stability within its training horizon, and TRC better preserved content and voice beyond it.Combining self-distillation with TRC also substantially reduced KV-cache growth.Source: §6 · p. 10
  • LLMs generated simulated user utterances and teacher-trajectory summaries and evaluated conversation quality, while generative AI also assisted with code, literature, translation, editing, and drafting.The authors state that these tools did not materially shape the research ideas or methodology and were not used to design experiments or interpret results; they reviewed AI-assisted material before inclusion.Source: AI USE STATEMENT · p. 11

A.1 MODELS AND SIMULATION SETTINGS

  • The evaluation used four released baseline models, fixed inference settings, and a simulator whose model identity and system prompt were withheld from both the user LLM and CQ judge.Interaction timing followed processed-audio logical time; model utterances were classified by timing relative to user speech.Source: §A.1 · p. 15

A.2 USER-SIMULATOR INSTRUCTIONS

  • Each episode paired a persona and topic, with three repetitions per combination; user messages included the instructions, dialogue history, and turn-specific request.For interrupted user turns, the history retained only text delivered before cutoff.Source: §A.2 · p. 17

A.3 METRIC DEFINITIONS AND AGGREGATION

The appendix defines the temporal metrics and aggregation procedures used to track responsiveness, spoken content, and voice consistency, then describes uncertainty handling and conversation-quality judging.

A.3.1 RESPONSIVENESS

  • FTO measures the gap or overlap between user-utterance end and model-response onset; RT tracks timeouts among resolved response opportunities, and TS counts sessions terminated after three consecutive timeouts.FTO excludes backchannels, interruptions, and non-responses; a backchannel alone does not resolve an RT opportunity as a response.Source: §A.3.1 · pp. 17–19

A.3.2 SPOKEN CONTENT

  • Conversation quality is scored by an LLM judge on a 1–5 scale, while WER compares native text output with offline ASR of generated speech.WER uses timestamp-based text–speech association and a fixed correction for potentially unspoken text suffixes in MiniCPM-o 4.5, which may also mask genuine deletion errors.Source: §A.3.2 · pp. 19, 21, 22

A.3.3 VOICE

  • Speaker similarity measures voice consistency against a reference embedding formed from retained speech chunks in the episode’s first minute.For multi-minute summaries, minute scores are duration-weighted within each episode and then averaged equally across episodes.Source: §A.3.3 · p. 22

A.4 VALIDITY AND UNCERTAINTY

  • Metrics are aggregated only when minimum data requirements are met, and uncertainty is estimated with 2,000 episode-level bootstrap resamples for 95% confidence intervals.Sampled episodes remain grouped during resampling; reporting thresholds apply to temporal curves and summary tables.Source: §A.4 · p. 24

A.5 CONVERSATION-QUALITY JUDGE PROMPT

  • The CQ judge receives timestamped user utterances and offline ASR transcriptions of model speech, grouped into one-minute intervals, and returns a rationale and integer score for selected intervals.Sentence timing is estimated in proportion to word counts, and a sentence is included in every interval overlapping its estimated span.Source: §A.5 · p. 27

A.6 TEN-MINUTE BASELINE RESULTS

The appendix provides per-minute baseline estimates and selected examples of how degradation appears in individual conversations.

A.6 TEN-MINUTE BASELINE RESULTS

  • Table 15 reports per-minute estimates and sample counts for four baseline models; its caption explains count notation, reporting-threshold dashes, and the speaker-enrollment reference minute.Source: §A.6 · p. 27

A.7 QUALITATIVE FAILURE CASES

  • Table 16 presents selected baseline cases of response timeouts, repetitive replies, and disagreement between native text and speech transcripts.The examples are described as illustrations of degradation patterns seen in individual conversations.Source: §A.7 · p. 27

A.8 TWENTY-MINUTE LONG-SESSION RESULTS

The appendix points to per-minute results for 20-minute sessions and documents TRC-related configurations and prompts.

A.8 TWENTY-MINUTE LONG-SESSION RESULTS

  • Table 17 reports per-minute results for LoRA and TRC in 20-minute sessions, with 90 episodes per model and condition.The caption says parentheses indicate sample counts, dashes indicate unmet reporting thresholds, and Ref. marks the first minute used for speaker enrollment.Source: §A.8 · p. 27
  • Figure 9 illustrates native and compact positional indices and layer dependencies, including KV access between raw and gist tokens.The example uses the configuration shown in Figure 3a.Source: Figure 9 · p. 28
  • The appendix lists model-specific temporal units, TRC window and tile sizes, content–control assignment rules, and training configurations.The supplied table captions identify these configuration topics but do not give their values.Source: Table 18 · p. 28, Table 19 · p. 28, Table 20 · p. 28
  • The summarization prompt asks for concise prose that preserves concrete conversational details, speaker attribution, corrections, agreements, and unresolved points.A separate teacher prompt follows a segment boundary and uses the base prompt and preceding-dialogue summary as placeholders.Source: §A.8 · p. 29, Figure 11 · p. 29, Figure 10 · p. 29

B METHOD DETAILS

The method adapts context handling and self-distillation to extended speech trajectories, with positional and layer-level rules for TRC. It also specifies teacher-trajectory construction, optimization, and evaluations of KV growth, latency, and information retention.

B.1 POSITIONAL ENCODING

  • RoPE uses native absolute indices for raw dialogue keys and compact indices for prefix and gist keys placed consecutively before the local raw context.Gist queries use compact positions while their native positions follow the raw positions of their tile.Source: §B.1 · pp. 29–30

B.2 LAYER COMPUTATION

  • Gist tokens acquire dialogue information through layer-wise keys and values; first-layer gist keys and values are excluded from raw-query attention, and final-layer gist updates are omitted.Gist attention and feed-forward updates occur in all layers except the last.Source: §B.2 · pp. 30–31

B.3 MODEL-SPECIFIC TRC SETTINGS

  • TRC window and tile sizes use each model’s native temporal units while keeping their durations approximately equal across models.Prefix contents differ by model, with initial dialogue frames retained for PersonaPlex and Raon-SpeechChat to account for attention sinks.Source: §B.3 · p. 31

B.4 TEACHER TRAJECTORY CONSTRUCTION

  • Teacher trajectories combine five personas with 20 topics distinct from diagnostic scenarios, use five segments, and branch to produce 400 trajectories per model.Two topics are held out for validation; mean trajectory durations range from 8.8 to 9.6 minutes across models.Source: §B.4 · pp. 33–34

B.5 SELF-DISTILLATION AND OPTIMIZATION

  • Training aligns recorded native inputs and teacher predictions with student positions, masks KL supervision at segment starts and forced decisions, and uses hidden-state supervision across audio-conditioning positions.The shared gist embedding and LoRA adapters have 8.4M trainable parameters for PersonaPlex and 7.7M for the other two models, approximately 0.1% of each backbone Transformer.Source: §B.5 · p. 34

B.6 TEACHER-TRAJECTORY PROMPTS

  • Teacher-trajectory prompts use a summarizer with temperature 0.3 and zero presence penalty, alongside revised user-simulator prompts for continuing across trajectory segments.The prompt materials also include persona and topic instructions for trajectory generation.Source: Figure 12 · p. 30, §B.6 · p. 34

C.1 KV STORAGE AND BACKBONE LATENCY

  • TRC grows KV storage by one gist position per completed tile, while Base grows with raw positions; latency measurements include backbone passes, cache updates, and gist computation.Analytical estimates cover KV tensor storage only; PersonaPlex’s Base growth rate applies only before its 3,000-position KV ring saturates.Source: §C.1 · p. 35

C.2 SINGLE-NEEDLE RETENTION

  • Single-needle retention uses 100 fact-question probes, tests recall at 0, 5, and 10 minutes, and scores exact reference-answer matches from native text output.Each probe runs on a separate copy of model state; incorrect and unanswered probes count as errors.Source: §C.2 · pp. 35–36, Table 23 · p. 36

C.3 HIDDEN-STATE SUPERVISION ABLATION

The PersonaPlex ablation tests hidden-state supervision by setting its loss weight to zero while holding other training and evaluation conditions fixed.

C.3 HIDDEN-STATE SUPERVISION ABLATION

  • The ablation sets λhid = 0 while keeping λcnt = λctl = 1, using the same teacher trajectories, data split, TRC settings, and training configuration.Each condition uses its lowest-validation-loss checkpoint and is evaluated in 20-minute sessions.Source: §C.3 · p. 36

D LIMITATIONS AND FUTURE WORK

The authors identify untested user behaviors, limited recall under high compression, and limited adaptation data as boundaries of the study.

D LIMITATIONS AND FUTURE WORK

  • Long-session stability under user-initiated interruptions and backchannels remains untested because the simulator did not initiate them.The authors propose adding these behaviors for a more comprehensive long-session assessment.Source: D LIMITATIONS AND FUTURE WORK · p. 36
  • Recall accuracy remained limited under high compression, leaving a compression–recall trade-off for future work.The authors suggest exploring objectives that improve retention and use of specific information.Source: D LIMITATIONS AND FUTURE WORK · p. 36
  • Limited adaptation data may have biased the models’ native full-duplex behaviors, and larger, more diverse datasets remain to be investigated.Source: D LIMITATIONS AND FUTURE WORK · p. 36

Related papers

Loading 2610.34991v1…