Source-linked AI summary
DuplexAgent-RSI: Recursive Harness Improvement for Full-Duplex Voice Agent Collaboration
Yingda Shen, Yuxiang Wang, Kunyu Feng, Qinke Ni, Jiaqi Li, Minghao Hsu, Junan Zhang, Dekun Chen, Yutong Bian, Zhizheng Wu
TL;DR
Live voice agents need to pair continuous conversation with extended delegated work, but existing coordination heuristics are difficult to diagnose and repair. DuplexAgent modularizes the harness, and Duplex-Harness-RSI uses simulated traces and paired checks to revise it; retained harnesses improve spoken knowledge, executable tool use, and fixed-core collaboration on VoiceChat and Qwen-Realtime.
Problem
Existing systems delegate demanding work from live voice interaction, but their coupled coordination heuristics make recurring failures difficult to isolate, evaluate, and transfer.
Source: §1 · p. 2Method
DuplexAgent uses six editable harness modules, while Duplex-Harness-RSI uses simulated failure traces, an Exam Planner, a Harness Editor, and paired checks to revise modules.
Source: §1 · pp. 2–3Results
DuplexAgent improves spoken knowledge, executable tool use, and fixed-core collaboration on VoiceChat and Qwen-Realtime.
Source: §7 · p. 14, §6 · p. 14Takeaways & Limitations
The results support a division in which the interaction model maintains the live conversation, the delegation pool supplies task capability, and the harness coordinates them.
Source: §6 · p. 14The loop edits harness instructions and executable rules while interaction-model weights, overlapped-speech perception, and turn-end judgment remain unchanged.
Source: §6 · p. 14
Abstract
from arXiv · showhide
Voice agents are converging on a collaboration pattern: a full-duplex interaction model stays on the live channel as the entry to the conversation, while search, reasoning, and coding are handled through asynchronous delegation. A duplex model supports continuous listening and speaking, but complex reasoning and tool use may exceed its capabilities. A coding agent can plan and execute extended tasks, but its sequential interface is a poor fit for live conversation. Combining them requires a harness that coordinates task acceptance, progress, cancellation, replacement, and result delivery while keeping the conversation responsive. Existing harnesses often rely on coupled heuristics, making them difficult to improve systematically from evidence. We present DuplexAgent, a full-duplex collaboration system whose harness expresses this workflow as six editable modules, and Duplex-Harness-RSI, a closed loop that revises them from interaction traces. A simulator automatically generates timed test conversations, runs the system, and produces failure traces that identify the collaboration modules requiring repair. Reasoning LLMs and coding agents in the delegation pool also serve the improvement loop: the Exam Planner selects the next tests from observed weaknesses and the repair archive, and the Harness Editor proposes targeted module changes. The capabilities that serve the user thus also improve the system's coordination. Experiments on intelligence, agentic, and duplex benchmarks show that DuplexAgent combines continuous interaction with difficult reasoning and complex task execution, achieving stronger spoken-knowledge and executable-tool scores than the compared delegated systems while maintaining strong interruption response. A harness ablation further shows that this modular, verifiable loop outperforms the initial harness and repeated editing that lacks its diagnosis and repair archive.
1 Introduction
The introduction motivates coordinating live conversation with delegated work, then presents a modular harness and a trace-driven loop for improving it. A simulator and targeted repairs make collaboration failures diagnosable and revisions testable.
1 Introduction
- Existing systems bridge live interaction and delegated work, but coupled coordination heuristics make recurring failures difficult to isolate, evaluate, and transfer.The harness must update or cancel tasks and return results at valid speaking opportunities while the conversation continues.Source: §1 · p. 2
- DuplexAgent keeps a full-duplex model as the conversational entry and assigns reasoning or coding work to delegates through six editable harness modules.Policy, Intake, Hold, Worker, Deliver, and Memory separate turn handling, task interpretation, pending work, delegation, result delivery, and shared context.Source: §1 · p. 2
- Separating modules lets traces distinguish an unupdated task from an obsolete result delivery and target Intake, Deliver, or both for repair.The proposed change can then be evaluated against the failed conversation and other tests.Source: §1 · p. 2
- Duplex-Harness-RSI reuses the delegation pool: an Exam Planner selects tests from failures and the repair archive, while a Harness Editor revises diagnosed modules.Paired candidate comparisons and a disjoint protection set determine whether a revision is retained.Source: §1 · pp. 2–3
- The simulator generates timed conversations across nineteen request types and twenty-two interaction patterns, supplying tests and failure traces for harness revision.Its pipeline covers content generation, temporal composition, speech rendering, and evaluation; it can also generate conversations for other interaction systems.Source: §1 · p. 3
2 Related work
Related work covers full-duplex speech and delegated computation, spoken-agent evaluation, and recursive harness improvement. DuplexAgent focuses on diagnosing and revising coordination for live spoken collaboration.
Full-duplex models and delegated agents
- Full-duplex speech research studies streaming understanding and generation, while related systems connect live interaction to asynchronous retrieval, background work, or long-horizon execution.DuplexAgent builds on this architectural direction and studies diagnosis and revision of coordination mechanisms.Source: §2 · p. 3
Evaluation of spoken agents
- Spoken-agent benchmarks measure knowledge, tool execution, and interaction, while the simulator adds task transitions and module-attributed failures for harness revision.External benchmark scores and simulator collaboration scores answer different questions and are reported separately.Source: §2 · p. 3
Recursive harness improvement
- Recursive agent-design work revises prompts, tools, or workflows from traces and scores; Duplex-Harness-RSI adapts tests and edits modules for live spoken collaboration.Its fixed evaluation boundary covers task identity, conversational continuity, and delivery timing.Source: §2 · p. 3
3 Method
DuplexAgent separates live interaction from asynchronous delegated work through six editable harness modules, and Duplex-Harness-RSI uses simulated conversations and trace-based diagnosis to revise that harness. Its tests cover request types, interaction patterns, and input conditions while keeping evaluation expectations fixed.
3.1 The collaboration harness
- The six-module harness coordinates task routing, interpretation, pending work, delegation, delivery, and shared context so failures can be attributed to bounded decisions.A task ledger tracks identity, version, status, and delivery history; late returns cannot make obsolete tasks current again.Source: §3.1 · pp. 4–5, Table 1 · p. 5
3.2 Composing test conversations
- Timed tests combine compatible request types and interaction patterns, fix expected behavior before adding spoken and acoustic variation, and produce executable evaluation expectations.Coverage spans nineteen request types and twenty-two interaction patterns; deterministic composition specifies task transitions, delegate delays or failures, and speaking windows.Source: §3.2 · p. 5, Figure 4 · p. 6
3.3 Trace-based diagnosis and collaboration score
- Trace-based diagnosis selects a terminal failure cause in priority order and uses intermediate decisions and returned payloads to attribute it to a harness module.Runs that fail only because the interaction model did not perceive input are excluded from Q, the percentage of successful conversations.Source: §3.3 · pp. 6–7
3.4 Duplex-Harness-RSI
- Duplex-Harness-RSI uses attributed failures and the repair archive to plan exams, propose targeted harness edits, and retain candidates only when they pass development and protection gates.The gates compare paired runs, preserve invariants, and limit regressions; protection traces remain hidden from the editor, and rejected candidates do not replace the incumbent.Source: §3.4 · pp. 7–8
4 Implementation
The implementation connects the interaction model, delegation pool, and harness through declared event and adapter interfaces. Experiments revise harness files while keeping model weights fixed.
4 Implementation
- The session engine processes speech, transcripts, and delegation or control signals, while adapters route instructions, context, and results according to declared capabilities.Adapter declarations distinguish capabilities such as function calling, textual delegation markers, context injection, and interim tool-output handling.Source: §4 · p. 8
- DeepSeek-v4-flash handles reasoning and planning, while Codex supplies coding capability and serves as the Harness Editor in an isolated workspace.The Exam Planner runs offline and reads the visible portion of the repair archive.Source: §4 · p. 8
- The editor can change only declared harness surfaces; the evaluator rejects undeclared paths and excludes scoring rules, simulator code, and scenario-specific answers from edits.The experiments use the released implementation’s default configuration and keep interaction-model weights fixed while revising the six harness modules.Source: §4 · p. 8
5 Experiments
The experiments compare delegated systems on external spoken-knowledge, executable-tool, and duplex-timing tasks, then evaluate harness revisions on simulator conversations. DuplexAgent improves tool and spoken-knowledge results, while timing remains attachment- and condition-dependent; the revision loop outperforms simpler editing approaches on the fixed core.
5.1 Evaluation setup
- Evaluation covers spoken multiple-choice knowledge, BFCL and FDB-v3 tool use, and streamed pause handling, turn taking, and interruption.Comparisons within each benchmark block hold the interaction model fixed; the two harness-revision campaigns keep model weights fixed and repeat the fixed core.Source: §5.1 · p. 9
5.2.1 Executable tool use
- BFCL average rises from 53.5 to 75.8 on VoiceChat and from 82.0 to 90.8 on Qwen-Realtime; DuplexAgent also improves FDB-v3 results on both attachments.On VoiceChat, FDB-v3 changes from 60.2 / 26.2 / 16.0 to 97.4 / 57.0 / 47.0; on Qwen-Realtime, Pass@1 rises from 39.0 to 49.0.Source: Table 4 · p. 10, §5.2.1 · p. 11
5.2.2 Spoken knowledge and duplex timing
- OBQA rises from 54.2 to 80.6 on VoiceChat and from 86.3 to 92.1 on Qwen-Realtime; MMSU rises from 36.1 to 80.5 and from 64.8 to 82.7.The table reports spoken-knowledge accuracy alongside duplex timing.Source: Table 5 · p. 11, §5.2.2 · p. 12
5.3 Harness improvement
- On the VoiceChat fixed core, r10 scores 85.3, 30.7 points above g0 and 23.4 above Sequential Editing.One-shot Editing scores 57.4 and Sequential Editing 61.9; Figure 6 compares the harnesses on the fixed core.Source: Figure 6 · p. 12, §5.3 · p. 13
5.2.2 Spoken knowledge and duplex timing
- Timing stays near the attached interaction model, but Qwen-Realtime synthetic pause TOR worsens from 0.074 to 0.147 while its turn and interrupt TOR improve.With VoiceChat, turn and interrupt TOR decline slightly from 0.983 / 1.000 to 0.961 / 0.965; Candor pause TOR improves from 0.278 to 0.231.Source: Table 5 · p. 11, §5.2.2 · p. 12
5.3 Harness improvement
- On Qwen-Realtime, eight retained versions raise the fixed-core score from 47.1 at g0 to 84.8 at g8, a 37.7-point gain.The retained-score staircase advances only when a candidate passes development and protection gates; proposals that fail can make the dashed series fall.Source: Figure 7 · p. 13, §5.3 · p. 13
- The fixed core shows missed delegation falling from 30.3% under g0 to 1.3% under Duplex-Harness-RSI, while extra delegation rises from 3.9% to 6.6%.Ignored control remains 5.3%.Source: §5.3 · p. 13
- Repairs target module-specific failures: missed cancellations fall 81%, missing progress updates 75%, and wrong answers 66.7% on their local exams.Deliver changes reduce overlaps and concealed failures by 100%, while assembling complete turns and retaining context reduces Memory-related failures by 60%; these are local-exam results.Source: §5.3 · p. 13, Table 6 · p. 14
6 Discussion and limitations
The results support a division in which the interaction model maintains conversation, the delegation pool supplies task capability, and the harness coordinates them. Harness revisions improve coordination, but do not change the interaction model’s perception or timing.
6 Discussion and limitations
- The interaction model maintains the live conversation, the delegation pool supplies task capability, and the harness coordinates the two.Retained repairs make routing, task control, context, and delivery timing more precise; spoken knowledge, executable tool use, and fixed-core collaboration rise with that precision.Source: §6 · p. 14
- Duplex-Harness-RSI rewrites module instructions and executable rules while keeping the interaction-model weights fixed.Module edits can impose requirements such as waiting for silence or a complete question, but do not alter the model’s perception of overlapped speech or its judgment that a turn has ended.Source: §6 · p. 14
- The remaining perception and timing failures motivate extending simulator-based improvement from the harness to the interaction model.The simulator can continue supplying exams, traces, and failure signatures for agentic reinforcement learning on the live timeline.Source: §6 · p. 14
7 Conclusion
DuplexAgent coordinates a live full-duplex interaction model and a delegation pool through six editable modules. On two interaction models, retained harness revisions improve spoken knowledge, executable tool use, and fixed-core collaboration, while perception and timing remain beyond module edits.
7 Conclusion
- On VoiceChat and Qwen-Realtime, the retained harness improves spoken knowledge, executable tool use, and collaboration on the fixed core.The interaction-model weights remain those used at the start of the campaign.Source: §7 · p. 14
- DuplexAgent expresses the exchange between a live full-duplex interaction model and a delegation pool in six editable modules.The modules are Policy, Intake, Hold, Worker, Deliver, and Memory.Source: §7 · p. 14
- Duplex-Harness-RSI uses timed simulator tests, attributed failures, and a repair archive to guide module revisions and paired checks.The Exam Planner selects the next conversations, and the Harness Editor revises modules named by diagnosis.Source: §7 · p. 14
- The revisions do not reach the interaction model’s perception and timing, which the paper identifies as a target for simulator-based agentic reinforcement learning.The model weights remain unchanged during the harness campaign.Source: §7 · p. 14