PaperlayerView exact revision on arXiv

Source-linked AI summary

DuplexCadence: Exact State and Execution from a Speech Model's Declared Timelines

Haixiao Gao, Yimin Zheng, Linyou Xiao, Zeke Xie

arXiv:2609.35115v1cs.AI

TL;DR

Full-duplex speech serving must meet a one-second cadence, but existing runtime techniques do not fit streaming state dynamics. DuplexCadence declares model-native clocks to coordinate demand-sized state allocation and exact-shape replay. It reports faster execution with lower peak memory and reduced live-path SPEAK time.

  • Problem

    Streaming history changes shapes and addresses, preventing graph recording, while runtime-invisible retention bounds prevent demand-sized allocation.

    Source: §1 · p. 2

  • Method

    DuplexCadence declares per-region native clocks and retention bounds, enabling demand-sized, address-stable state allocation together with exact-shape replay without padding.

    Source: §1 · p. 2

  • Results

    DuplexCadence achieves 2.85× speedup and 38.8% lower peak memory; mean SPEAK latency shifts from 14% above cadence to 2% below it.

    Source: §1 · p. 3, §6 · p. 13

  • Takeaways & Limitations

    The authors frame retention boundedness and clock declarability as real-time-serving principles parallel to batchability in LLM serving.

    Source: §6 · p. 13

    Exact specialization is bounded by catalog width, and cache-free prompt priming and turn-boundary pointer misses still execute eagerly.

    Source: §4.6 · p. 12

Abstract

from arXiv · show

Full-duplex speech models support streaming interaction that listens and speaks at the same time. Serving them is governed by a strict, repeating deadline: conversation advances on a one-second cadence, and every second of input must be turned into a second of speech before the next second arrives. Because stages within a session run in strict sequence, per-invocation overhead cannot be batched away. Profiling reveals that the autoregressive stages of a duplex second already fit within the period, whereas the token-to-audio synthesis tail is what causes overruns. This tail stage suffers from orchestration slack where the GPU is left waiting as thousands of tiny, regular operations are issued one by one, while also wasting substantial memory by over-provisioning state at static implementation constants. Existing remedies, such as graph recording and demand-sized allocation, fail because streaming state dynamics violate their prerequisites. The root cause is that the runtime lacks the model's native clocks: the per-region counters that govern advancement rates and retention policies. We propose DuplexCadence, which explicitly declares native clocks to the runtime and derives two mutually enabling rules: demand-sized state allocation at a stable address, and exact-shape graph replay without padding. The former eliminates idle memory and stabilizes tensor pointers, while the latter removes orchestration slack without padding overhead. Evaluated on four released models across three decoder architectures with bit-for-bit identical output, DuplexCadence reaches $2.85\times$ the stock runtime's speed at $38.8\%$ lower peak memory. On the live duplex path, mean SPEAK time falls from $14\%$ over the one-second cadence to $2\%$ under it, enabling models to reliably keep up with interactive speech while markedly expanding multi-

1 Introduction

Duplex speech serving must meet a one-second deadline, but sequential stages prevent batching within a session. DuplexCadence exposes model timelines so the runtime can coordinate demand-sized state and exact-shape replay.

1 Introduction

  • A full-duplex system must turn each second of input into a second of speech before the next second arrives, but sequential stages leave no within-session batching target.Per-invocation launch overhead therefore lands on the single session’s critical path.Source: §1 · p. 1
  • The synthesis tail takes about 1.17 seconds per second of work, with the GPU waiting for 57–65% of wall-clock time and the flow decoder using only one-seventh of allocated workspace.Launch overhead threatens single-stream cadence; unused memory limits concurrent sessions.Source: §1 · pp. 1–2
  • Streaming history changes tensor shapes and pointers, so graph recording’s fixed-shape and stable-address requirements fail, while hidden retention bounds prevent demand-sized allocation.The stock CUDA Graph path consequently falls back to individual operator launches.Source: §1 · p. 2
  • DuplexCadence declares per-region native clocks and retention bounds to jointly enable demand-sized, address-stable state and exact-shape replay without padding.The two rules are mutually enabling; byte-for-byte equality is checked before every invocation.Source: §1 · pp. 2–3
  • On the streaming speech synthesis module shared by MiniCPM-o 4.5 and Lychee-FD, DuplexCadence achieves 2.85× stock-runtime speed and 38.8% lower peak memory, with results gated on byte-for-byte equality.Source: §1 · p. 3
  • The native-clock contract is proposed as a general principle for sequential, strict-cadence workloads beyond speech, including streaming translation, video generation, and robotic control.In these settings, fixed launch overhead cannot be amortized across requests.Source: §1 · p. 3

2 Background and Motivation

Interactive sessions advance on a one-second cadence, and sequential dependencies prevent intra-session batching; bounded retention and fixed timeline rates define the workload class. Profiling exposes coupled execution and memory inefficiencies that existing tools do not resolve independently.

2.1 Multi-Clock Real-Time Streaming Runtimes

  • The workload class has fixed-rate clocks, at least one retention-bounded persistent region, and sequentially dependent stages within one interactive session.Offline TTS and unbounded LLM serving fall outside this scope; bounded-cache systems join when their chunk policy is fixed.Source: §2.1 · p. 3

2.2 Anatomy of a Full-Duplex Second

  • 327 ms of synthesis after 842 ms of earlier stages pushes a duplex second 16.9% past deadline; the 2.85× speedup targets the synthesis tail.The autoregressive fraction is outside the lossless scope.Source: §2.2.1 · p. 4
  • Orchestration slack reaches 57–65% of eager wall-clock time, while persistent-state over-reservation ranges from 3.5× to 18× across regions.Concatenation moves cache tensors to new addresses; together, memory over-reservation and orchestration slack are coupled gaps.Source: §2.2.1 · p. 4, Table 2 · p. 5, §2.2.2 · p. 5

2.3 Why Existing Tools Do Not Close the Gap

  • Hand-picked graph buckets do not match native-clock shapes, while padding preserves over-reservation; graph recording alone or compaction alone leaves the other inefficiency unresolved.The paper also reports that torch.compile aborts on key streaming paths and introduces numerical drift while saving no memory.Source: §2.3 · p. 5
  • Declaring native clocks makes reachable shapes and live history extents knowable, enabling demand-sized state at stable addresses and exact-shape replay without padding.Declared retention policies bound live history; deterministic clocks and configuration-bound shapes make reachable shapes knowable before inference traffic.Source: §2.3 · p. 5

3 The Native-Clock Contract and Its Realization

DuplexCadence turns model-declared clocks and retention policies into a contract from which the runtime derives demand-sized state and exact-shape graph replay. The coupled rules target memory overhead and launch slack while preserving bit-for-bit execution, within a lossless specialization scope bounded by reachable signatures.

3 The Native-Clock Contract and Its Realization

  • The adapter declares clocks, retention policies, and address-stable regions; the runtime derives state sizing and exact-signature graph capture from this contract.A region contract records its clock, growth, lifetime, mutability, retention policy, and placement.Source: §3 · p. 6, §3.1 · p. 6

3.2 State Rule: Demand Sizing and the Carry

  • Demand sizing keeps the carry at a stable address and allocates state to clock- and retention-implied extents rather than implementation constants.The solver workspace shrinks from a 16×1000 envelope to 10×458; the carry holds retained history at one address.Source: §3.2 · p. 6, §3.1 · p. 6
  • The carry also fits distinct decoder state designs: it becomes Moshi’s ring buffer and Freeze-Omni’s compact convolutional left-context buffer.The paper presents these as applications of the same contract.Source: §3.2 · p. 7

3.3 Compute Rule: Exact-Shape Replay

  • Exact-shape replay captures CUDA graphs for reachable signatures, with runtime admission checks and eager fallback when a signature or pointer misses.Under the stated certification conditions, replay reproduces eager state transitions and outputs bit for bit.Source: §3.3 · p. 7

3.4 Why the Two Rules Are Inseparable

  • Demand sizing and zero-copy aliasing jointly reduce replay memory overhead; demand sizing alone leaves launch count and orchestration slack unchanged.Time-multiplexing graph bindings across sessions means additional sessions replicate only lightweight state snapshots.Source: §3.4 · p. 8

3.5 Scope: Catalog Width

  • Exact specialization requires a compact catalog: bounded retention alone is insufficient when extent advances in small strides.Unit-stride retention yields K_r = W_r + 1, and tabled failure modes include a language backbone with K_r = 2,001 and speech-token queries with traffic-dependent lengths.Source: §3.5 · pp. 8–9, Table 3 · p. 9

3.6 What If the Declaration Is Wrong

  • The contract’s premises are checked at runtime or capture, and a class that fails certification is not admitted.A shadow audit caught an append-ordered carry that diverged on chunk three after cache eviction reached the prompt tail.Source: §3.6 · p. 9

4 Evaluation

The evaluation tests whether the state and compute rules work together, transfer to the live duplex path and other architectures, and compare favorably with compiler and vendor-graph controls. Across these settings, it measures latency, memory, exactness, and serving capacity under matched workloads.

4.1 Setup

  • The evaluation uses matched blocks with shared seeds and inputs, fresh processes, paired intervals, and exactness gates against the unmodified baseline.Primary results run on an A100; the factorial, serving points, and end-to-end MiniCPM-o are also run on an L20.Source: §4.1 · pp. 9–10, §4 · p. 9

4.2 Evidence for Mutual Enabling

  • With both rules, SRc reaches −64.8% latency and −38.5% peak memory across 40/40 bit-exact blocks; neither single-rule variant achieves both gains.The state rule alone has an uncertain −0.5% latency estimate, while replay alone incurs memory costs; coarsening helps latency and memory only when the state rule is active.Source: Table 4 · p. 10, §4.2 · p. 10

4.3 Surviving the Full Duplex Path

  • In the live MiniCPM-o 4.5 path, synthesis falls from 201.7 to 72.2 ms, reproducing the standalone 2.85× speedup to within 3%; mean SPEAK time falls from 1,141 to 976 ms.All six matched blocks preserve token-ID sequences, float32 PCM traces, control decisions, and turn boundaries exactly; mean SPEAK time moves from 14.1% over to 2.4% under the one-second cadence.Source: §4.3 · pp. 10–11, Table 5 · p. 11
  • At shared-weight N=4, DuplexCadence completes every speak-heavy block within ≈26 GiB while the baseline runs out of memory in all six blocks.Peak allocation falls in all 24 matched pairs, by 6.0 GiB at mixed N=2 and 9.3 GiB at N=4; PCM byte equality is 100%.Source: Figure 4 · p. 11, §4.3 · p. 11

4.4 Cross-Architecture Evidence

  • On Moshi, declarative ExactShapeReplay recovers 3.46× speedup with 3/3 bitwise equality, capturing 89% of the custom optimization’s reduction.On an L20, per-chunk speedup reaches 2.95× alongside a 38% peak-memory reduction, and both single-rule falsifiers reproduce.Source: §4.4 · p. 11, Table 6 · p. 11, §4.5 · p. 12

4.5 Compiler and Vendor-Graph Controls

  • Compiler controls either fail or lose key properties: Inductor aborts on the diffusion loop, corrupts streaming-encoder graph outputs, and its generated kernels remain 38.7% slower with divergent bytes.Re-bucketing the vendor graph reaches −52% latency versus eager, but remains 43.3% slower and 73% larger than SRc; no control achieves speedup, memory reduction, and bitwise exactness together.Source: §4.5 · pp. 11–12, Table 7 · p. 12

4.6 Limitations

  • Specialization is limited by catalog width, and cache-free prompt priming and turn-boundary pointer misses still execute eagerly.The shared-weight harness uses a 2 s window with wide goodput intervals; continuous cross-session composition remains open, and cross-process reproducibility depends on the deterministic kernel library.Source: §4.6 · p. 12

5 Related Work

DuplexCadence distinguishes its intra-session execution approach from serving methods that batch independent requests, while connecting native-clock execution to deterministic replay and bounded streaming state.

Serving systems

  • Cross-request batching and serving techniques amortize launch costs across independent requests, whereas DuplexCadence targets orchestration slack within individual stages.Nova pipelines stages within a request to mitigate inter-stage latency; DuplexCadence instead eliminates orchestration slack within stages.Source: §5 · p. 12

Predictable execution

  • Unlike timing inferred from profiles or dataflow graphs, DuplexCadence uses declarative configuration timing to support strict bitwise equality.Clockwork schedules inference timing proactively, while Nimble and Rammer reorder GPU execution to reduce launch overhead.Source: §5 · p. 12

Shape specialization and bounded retention

  • DuplexCadence enumerates reachable shapes exactly rather than relying on padding or polymorphic dispatch, preserving numerical reductions.Its retention bounds come from configuration and are applied without eviction, enabling exact-shape replay.Source: §5 · p. 12

Approximation techniques and speech models

  • Unlike approximation techniques that modify computation and risk bitwise equality, DuplexCadence treats model temporal rates as an execution contract.The paper positions this systems optimization as operating alongside work on speech-model representation and perceptual quality.Source: §5 · p. 12

6 Conclusion

DuplexCadence declares model timelines to enable demand-sized state allocation at stable addresses and exact-shape graph replay. It improves speed and memory use while bringing live SPEAK latency below the one-second cadence.

6 Conclusion

  • 2.85× speedup and 38.8% lower peak memory versus the stock runtime, with bit-identical outputs.The two rules are demand-sized state allocation at a stable address and exact-shape graph replay.Source: §6 · p. 13
  • Mean SPEAK latency falls from 14% above cadence to 2% below it on the live duplex path.The paper reports that this enables models to keep up with interactive speech and expands multi-session capacity.Source: §6 · p. 13
  • The authors propose retention boundedness and clock declarability as real-time serving principles alongside batchability in LLM serving.They state that optimization scope is computable ahead of time.Source: §6 · p. 13

Related papers

Loading 2609.35115v1…