DuplexSLA: A Full-Duplex Spoken Language Model with Synchronized Speech, Language, and Action

1 Introduction

• Turn-based speech pipelines struggle to distinguish pauses, backchannels, and interruptions, while tool calls before or after speech add delay and mid-utterance calls can disrupt speech.An external semantic VAD adds latency and does not see the assistant’s internal state.Source: §1 · pp. 1–2
• DuplexSLA combines native interruption, pause, and backchannel control with in-conversation planning and tool calling alongside assistant speech.These capabilities are co-decoded by the same backbone on a shared 160 ms chunk timeline.Source: §1 · p. 2
• The action channel gives time-stamped tool calls and turn-taking decisions a dedicated textual lane co-decoded with assistant audio, rather than a post-hoc cascade or competing for speech-text slots.It can carry planning text and structured tool calls while the assistant continues speaking.Source: §1 · p. 2
• DuplexSLA-Bench contains 2,100 cases and evaluates pause, interrupt, and backchannel turn-taking alongside timing-aware tool calling.The reported evaluation finds sub-second latency while remaining competitive on tool-call accuracy.Source: §1 · p. 2

2.2 Per-chunk serialization

• Each 160 ms chunk serializes user audio, assistant TA4, and action text; aligning generated speech and actions to that chunk gives tool calls timestamps without disturbing assistant audio.The action text can carry planning, control labels, or structured tool calls.Source: §2.1 · p. 3, §2.2 · p. 4

2.1 Dual-stream three-channel formulation

• The three-channel interface combines continuous user audio, discrete assistant speech, and a text-only action stream on the assistant timeline, decoded jointly with the next assistant-audio chunk.The assistant channel uses a text anchor followed by four audio tokens per chunk.Source: §2.1 · p. 3

2.3 Real-time decoding budget

• The action channel is capped at 10 text tokens per chunk to fit the 160 ms decoding budget; excess tokens spill into following chunks, and the cap can be retuned per accelerator without retraining.The authors describe this as a deployment budget, not an architectural constraint.Source: §2.3 · p. 4

2.4 Native interruption, pause and backchannel

• Shared-backbone control handles pauses with silence anchors, keeps speaking through backchannels, and yields on interruptions within small chunk-level latency.The action channel carries interaction-control labels while assistant speech continues or switches to silence as appropriate.Source: §2.4 · pp. 4–5, Figure 2 · p. 5

2.5 In-conversation planning and tool calling

• Planning fragments and structured calls share the action channel, enabling backchannel-triggered tools during speech and multiple request-ordered calls alongside ongoing audio.The examples include a music call during assistant speech and three time-aligned calls for one user turn.Source: §2.5 · pp. 5–6, Figure 3 · p. 6

2.6 Action channel design rationale

• A separate action channel avoids mixing tool-call JSON with assistant audio tokens, provides timestamps for downstream execution and latency evaluation, and costs at most 10 tokens per chunk.The authors state that this modest cost fits within the per-chunk decoding budget.Source: §2.6 · p. 6

3.2 Mixture proportions

• CPT combines duplex dialogue, dual-side ASR, and text, whereas smaller post-training concentrates on interaction control and three tool-call styles.Figure 5 shows the stage distribution; Table 2 reports the concrete scales.Source: Figure 5 · p. 8, §3.2 · p. 8, Table 2 · p. 9

3.1 Sample form

• Each sample aligns user and assistant audio with ordered, trigger-timed actions, while supervising assistant audio and actions but only observing user audio.Across task families, the action channel carries delayed transcripts, control labels, or planning text and tool calls.Source: §3.1 · pp. 7–8

3.3 Action objects on the timeline

• A trigger-time FIFO serializes simultaneous actions and spills over-budget calls across chunks without preempting earlier calls or blocking assistant audio.Tool calls stay in the user’s request order without delaying the spoken response.Source: §3.3 · pp. 9–10

3.4 Dual-side ASR is required for time alignment

• Dual-side ASR ties assistant transcript characters to their spoken chunks, aligning audio and action timing to support sub-second tool-call latency.Source: §3.4 · p. 10

4.1 Stage 1: continued pretraining

• CPT’s ∼500k audio hours include duplex dialogue and 2 × 90k h of dual-side ASR to teach serialization and timing, but do not yet produce targeted interaction behaviours.General text data supplements the audio mixture to preserve language and reasoning ability.Source: §4.1 · p. 10

4.2 Stage 2: capability-oriented post-training

• Post-training uses ∼50k hours, including ∼36k h for pause, interrupt, and backchannel control and ∼14k h for three tool-call patterns.Tool-call data covers single-action, multi-action, and backchannel-action requests.Source: §4.2 · pp. 10–11

4.3 Loss with full-duplex-aware masking and reweighting

• Training combines next-token cross-entropy on assistant TA4 and action streams with text-only modeling, then masks and reweights selected tokens using one decoder interface across stages and tasks.Source: §4.3 · p. 11

4.4 Stage division by capability

• The authors report the staged division as their most data-efficient setup: dialogue-only misses interaction control, while capability-heavy training from the outset reduces assistant speech smoothness.Source: §4.4 · p. 11

5.1 Benchmark and evaluation protocol

• The benchmark includes 1,200 turn-taking cases and 900 tool-call cases, and scores turn-taking accuracy plus delay from semantic anchors.Tool-call cases require matching function names and arguments, with trigger times inside a specified window.Source: §5.1 · p. 11, §5.2 · p. 12, Table 3 · p. 12

5.2 Tool-call results

• DuplexSLA matches the cascade in tool-call accuracy while delivering ∼4x lower delay on average.The benchmark covers single-action, multi-action, and backchannel-action requests.Source: §5.2 · p. 12, Table 5 · p. 13

5.3 Full-duplex turn-taking results

• In context-prefill tests, DuplexSLA has the highest accuracy and lowest delay across all four turn-taking scenarios.It handles backchannel correctly at 98.33%, versus ≤40% for every baseline; closed-source baselines expose no backchannel label.Source: §5.3 · p. 13, Table 6 · p. 13
• Without context prefill, DuplexSLA is the only sub-second system with competitive accuracy, averaging 94.34% while reaching ∼0.30 s delay.Open-source duplex backbones without targeted post-training collapse on the pause subset.Source: §5.3 · p. 13, Table 7 · p. 13

5.4 Take-aways

• Across the tables, DuplexSLA combines sub-second turn-taking responses with cascade-matching tool-call accuracy at ∼4x lower delay.The authors attribute tool-call timing to emitting planning and calls without waiting for a turn boundary or interrupting assistant audio.Source: §5.4 · p. 14

6 Conclusion

• DuplexSLA places listening, speaking, planning, and tool calling on a shared 160 ms chunk timeline.Its channels are continuous user audio, discrete assistant TA4, and rate-limited textual actions.Source: §6 · p. 14
• The model learns semantic-driven interruption, pause, and backchannel handling, plus in-conversation planning and tool calling alongside assistant speech.The conclusion describes these capabilities as learned by a single backbone using the dual-stream three-channel formulation.Source: §6 · p. 14
• On DuplexSLA-Bench, DuplexSLA achieves sub-second latency in all four turn-taking scenarios and competitive tool-call accuracy at 3−4x lower latency than an ASR + LLM cascade.The conclusion expects the action-channel design to extend to richer planning signals, multi-turn workflows, and broader open-domain spoken tool use.Source: §6 · p. 14

A Per-Chunk Serialization Case Studies

• Each appendix trace row represents one 160 ms chunk, with user audio, assistant TA4, and any action segment shown in sequence.The action segment appears after assistant TA4 and before the chunk-ending marker.Source: A Per-Chunk Serialization Case Studies · p. 18

A.1 User-channel ASR

• User-channel transcript text is emitted on the action channel with a fixed two-chunk lag, and same-chunk tokens are merged.The lag is 320 ms.Source: §A.1 · p. 18

A.2 Assistant-channel ASR

• Assistant text anchors are left-aligned and lack exact timing, so assistant transcripts are re-emitted on the action channel at the chunk where each character is spoken.This teaches alignment between assistant audio time and action time.Source: §A.2 · p. 18

A.3 Duplex dialogue with planning + tool call

• When a car-temperature request becomes clear, DuplexSLA emits planning text and a set_car_setting call while assistant audio continues.The ≤10-token chunk cap spreads the planning and call body across chunks, while the first action token remains aligned to the semantic anchor.Source: §A.3 · p. 19

A.4 An illustrative interrupt case

• When an acknowledgement becomes a new statement, the action channel emits an interrupt label and assistant audio switches to silence anchors.The expected response is to keep speaking through the acknowledgement, then yield with small chunk-level latency once new content begins.Source: §A.4 · p. 20

A.5 An illustrative backchannel case

• When the user gives a brief acknowledgement and then stays silent, DuplexSLA labels it a backchannel while continuing its planned response.The example contrasts this with an interruption, where new user content follows the acknowledgement.Source: §A.5 · p. 20

• Training data paraphrases the same control name with near-synonyms to reduce overfitting to one surface form.Source: B Action vocabulary and… · p. 21
• The control-label set is kept compact so turn-taking decisions are decoupled from spoken content.Source: Table 8 · p. 21
• The per-task system prompt supplies the task signal and fills its speaker placeholder with one of 18 canonical assistant speakers.Source: Table 9 · p. 21, B Action vocabulary and… · p. 21

• 2,100 cases are organised into two subsets that exercise DuplexSLA’s two highlighted capabilities.Source: C DuplexSLA-Bench composition and tool… · p. 21, Table 10 · p. 22
• The tool-call subset covers 50 cabin and smart-home functions spanning car control, navigation, media, and on-device search and queries.Source: Table 11 · p. 22
• Examples include UI scrolling, route planning, media playback, and weather queries.Source: C DuplexSLA-Bench composition and tool… · p. 23

• The per-chunk action-token cap is a deployment budget, not an architectural constant.Source: D Inference and serving notes · p. 24
• Each chunk emits five assistant TA4 tokens and up to ten action text tokens.Source: D Inference and serving notes · p. 24
• Causal speech encoding advances a chunk without future user audio; action tokens beyond ten spill into later chunks while the trigger remains anchored to the planning-text start.Tool-call closing markers may therefore land in a later chunk than the opening marker.Source: D Inference and serving notes · p. 24

• The shared conceptual action-object schema has a function name, optional planning text, optional parameters, and an offset.Source: E Action object schema · p. 24
• The name can identify one of 50 cabin and smart-home tools or a response, interrupt, backchannel, or ASR control label.Source: E Action object schema · p. 24
• Planning is short natural-language text, while parameters are JSON-style arguments and are empty for ASR or control-only labels.Source: E Action object schema · p. 24
• Actions are queued by offset and serialized in trigger-time order; tokens beyond the ten-token chunk budget spill forward while later-triggered actions wait.Source: E Action object schema · p. 24