SteerablePlex: Can We Steer Full-Duplex Models?

1. INTRODUCTION

• A simulator must adopt a prescribed persona, pursue goals, and create the situations an evaluation intends to probe; scenario drift can change the test interaction.Needed behaviors may include initiating a request, withholding information, or correcting itself.Source: §1 · p. 1
• GPT-Realtime follows most individual instructions but is unreliable at completing the full sequence in order, while current open-source full-duplex models exhibit a larger gap.SimIF-Bench measures topic adherence, naturalness, stage completion, and stage order during interaction.Source: §1 · p. 1
• Continuous listening and speaking leave no clear adjustment boundary, while system-prompt adherence can weaken with dialogue history and latency budgets constrain model size and long-horizon planning.These limitations motivate the question of whether full-duplex models can be steered at runtime.Source: §1 · p. 1
• SteerablePlex accepts instructions during a conversation, and an asynchronous text-LLM director monitors context and issues instructions when needed.The model uses a special-token-delimited interface and GDPO to jointly optimize instruction following, naturalness, and turn taking.Source: §1 · p. 1

2. SimIF-Bench

• SimIF-Bench evaluates topic adherence, naturalness, conversation goal completion, and exact four-goal order in staged examiner–examinee conversations.It reuses the 200 scenarios and interactive simulation setup of Full-Duplex-Bench v2, fixing PersonaPlex as the examinee.Source: §2 · p. 2
• Calls end after 120 s, and transcripts are assessed through three LLM-judging passes with transcript evidence required for every decision.Categorical decisions use majority vote, naturalness uses the median, and adherence and naturalness are scored on 1–5 scales.Source: §2 · p. 2
• The benchmark treats scenario focus despite unexpected partner behavior and natural interaction as requirements alongside ordered stage completion.A reliable simulator should also steer the interaction without sacrificing conversational naturalness.Source: §2 · p. 2

3. METHODOLOGY

• An asynchronous text-LLM director tracks task, conversation, and goals while a continuously running full-duplex actor handles speech, listening, and turn taking.Every 2 s, the director updates goal state and generates a candidate instruction and action.Source: §3 · p. 2

3.1. Instruction Fusion

• Runtime instructions are spliced into the model-text stream between special start- and end-of-instruction tokens, while both audio streams receive silence tokens.Conversation time does not advance during inserted instruction steps.Source: §3.1 · p. 2

3.2. Data Preparation

• Training data includes 9,417 Fisher instruction examples and 2,000 synthetic overlapping-speech dialogues built from benchmark scenarios.The Fisher examples total 130.8 hours of rollout time; the synthetic corpus contains 84.1 hours of stereo audio, with 200 dialogues held out.Source: §3.2 · p. 2

3.3. Instruction Following Training

• The pretrained model can follow an injected instruction when it is phrased as a short imperative, providing an initial capability for direct reinforcement learning.Source: §3.3 · p. 2
• GDPO independently normalizes reward dimensions before combining them, while training optimizes goal completion alongside naturalness and turn-taking rewards.The reward design addresses shortcuts in which the model blurts out requested content or talks over the partner.Source: §3.3 · p. 3
• Synthetic-dialogue rollouts preserve preceding conversation and instructions, insert each current instruction at its annotated turn, and cover four ordered goals per dialogue.Each synthetic dialogue yields one rollout per goal.Source: §3.3 · p. 3

4.1. Experimental Setup

• Both variants initialize from PersonaPlex and train LoRA adapters with GDPO; task-independent training adapts β toward a KL target, whereas in-domain training fixes β = 0.01.Both use learning rate 10^-5; the task-independent variant adapts β from 0.05 toward KL target 0.3.Source: §4.1 · p. 4

4.2. Baseline

• PersonaPlex has the highest naturalness but the lowest completion and exact-order scores, while GPT-Realtime remains the strongest baseline yet often fails to complete goals in order.KAME improves completion over unsteered PersonaPlex but sacrifices naturalness; Raon also illustrates the difficulty of completing all goals in order under dynamic interaction.Source: §4.2 · p. 4
• Task-independent SteerablePlex closes much of PersonaPlex’s completion gap and outperforms other open-source systems with a better completion–naturalness balance, but its exact-order score remains low.The authors say this is consistent with limited transfer from single-instruction training to multi-instruction streams.Source: §4.2 · p. 4
• SteerablePlex (in-domain) surpasses GPT-Realtime in completion and exact order, with the advantage also holding on 20 scenarios excluded from training.The authors associate this result with training on multiple ordered instructions within each dialogue.Source: §4.2 · p. 4

4.3. Turn-Taking Ability

• KAME’s continuous steering degrades all three backchannel measures and reduces interruption success and quality, making the model harder to interrupt.The authors suggest constant steering can weaken a full-duplex front end’s native turn-taking policy.Source: §4.3 · p. 4
• Both SteerablePlex variants improve five named turn-taking measures relative to PersonaPlex and gain on six of ten reported metrics, while backchannel distribution stays close to the base model’s.The remaining regressions are concentrated in response latency and backchannel frequency.Source: §4.3 · p. 4

• SimIF-Bench evaluates instruction following in dynamic full-duplex conversations.Source: §5 · p. 4
• SteerablePlex is a full-duplex front end trained with GDPO to follow runtime text instructions while preserving turn taking.Source: §5 · p. 4
• Coupled with an asynchronous director, SteerablePlex provides a path toward controllable, scenario-specific full-duplex user simulation.Source: §5 · p. 4