Source-linked AI summary
Game-Time: Evaluating Temporal Dynamics in Spoken Language Models
Kai-Wei Chang, En-Pei Hu, Chun-Yi Kuan, Wenze Ren, Wei-Chih Chen, Guan-Ting Lin, Yu Tsao, Shao-Hua Sun, Hung-yi Lee, James Glass
TL;DR
Conversational SLMs need to manage timing and simultaneous speech, but existing benchmarks do not directly evaluate temporal dynamics. Game-Time assesses instruction-following with temporal constraints, and its evaluation finds that nearly all models falter under those constraints. The authors report that the benchmark’s dual-channel evaluation aligns with human preferences.
Problem
Existing SLM benchmarks emphasize content, style, dialogue behavior, and turn-taking rather than temporal dynamics, which are important for real-time speech interaction.
Source: §1 · p. 1Method
Game-Time evaluates spoken instruction-following by formalizing each instance as a base task that must be performed while satisfying a set of constraints.
Source: §3.2 · p. 3Results
Many SLMs struggle with basic instruction-following, and nearly all evaluated models degrade when temporal constraints are introduced, especially for time awareness and real-time full-duplex capability.
Source: §1 · p. 2, §6 · pp. 4–5Takeaways & Limitations
The authors present the benchmark and its dual-channel evaluation as a way to measure temporal behaviors and motivate research on time-aware SLMs.
Source: §6 · p. 5The instructions prioritize natural language and time-cue interpretation over unequivocally precise commands, so evaluation depends on interpreting natural instructions.
Source: §4 · p. 3
Abstract
from arXiv · showhide
Conversational Spoken Language Models (SLMs) are emerging as a promising paradigm for real-time speech interaction. However, their capacity of temporal dynamics, including the ability to manage timing, tempo and simultaneous speaking, remains a critical and unevaluated challenge for conversational fluency. To address this gap, we introduce the Game-Time Benchmark, a framework to systematically assess these temporal capabilities. Inspired by how humans learn a language through language activities, Game-Time consists of basic instruction-following tasks and advanced tasks with temporal constraints, such as tempo adherence and synchronized responses. Our evaluation of diverse SLM architectures reveals a clear performance disparity: while state-of-the-art models handle basic tasks well, many contemporary systems still struggle with fundamental instruction-following. More critically, nearly all models degrade substantially under temporal constraints, exposing persistent weaknesses in time awareness and full-duplex interaction. The Game-Time Benchmark provides a foundation for guiding future research toward more temporally-aware conversational AI. Demos and datasets are available on our project website https://ga642381.github.io/Game-Time.
1. INTRODUCTION
Conversational SLMs must manage timing while listening and speaking in real time, but existing benchmarks do not directly assess temporal dynamics. Game-Time evaluates basic instruction-following and performance under temporal constraints, where many models still struggle.
1. INTRODUCTION
- Real-time full-duplex conversation requires models to listen and speak simultaneously while controlling what to say and when; precise timing is also relevant to time-critical instructions and synchronized user activities.The paper cites emergency guidance and healthcare voice agents as examples of these deployment needs.Source: §1 · p. 1
- Existing SLM benchmarks assess content, style, dialogue behaviors, and turn-taking, but not temporal dynamics; Game-Time targets perceiving, predicting, and producing speech in sync with users.The benchmark reframes evaluation around when to speak as well as what to say.Source: §1 · pp. 1–2
- Game-Time pairs basic instruction-following tasks with advanced tasks that add timing and synchronicity constraints, inspired by language activities through which children learn timing and tempo.The introduction uses rock-paper-scissors, where players act on a shared cue, as an example.Source: §1 · p. 1
- Many contemporary SLMs struggle with basic instruction-following, and nearly all evaluated models perform worse when temporal constraints are introduced.The paper identifies time awareness and real-time full-duplex capability as particular weaknesses.Source: §1 · p. 2
2. RELATED WORKS
Prior work has developed full-duplex SLM architectures and benchmarks for conversational behavior. The paper distinguishes those efforts from direct evaluation of explicit temporal constraints.
2.1. Full-duplex Spoken Language Models
- Full-duplex SLMs use either dual channels for concurrent listening and speaking [17] or time-multiplexing with a speak-or-wait mechanism [29] [18]; Game-Time compares both designs and commercial agents.Dual-channel designs support concurrent listening and speaking but increase modeling complexity; time-multiplexing systems switch to generation based on turn-taking cues.Source: §2.1 · p. 2
2.2. Benchmarks for Conversational Spoken Language Models
- Prior benchmarks assess understanding, paralinguistics, interruption, overlap, turn-taking, responsiveness, noise robustness, and audiovisual backchanneling, but do not directly test explicit temporal constraints.Game-Time focuses on whether models understand and satisfy such constraints.Source: §2.2 · p. 2
3. GAME-TIME BENCHMARK
Game-Time organizes spoken instruction-following tasks into basic and temporally constrained forms, formalizes them as tasks with constraints, and evaluates model outputs with an LLM judge. Its dataset uses diversified spoken instructions, and the judging approach is checked against human preferences.
3.1. Task Families
- The benchmark groups tasks into six basic families, spanning sequence recitation, repetition, composition, recall, open-ended conversation, and role-play.These tasks range from structured behaviors to open-ended and social behaviors.Source: §3.1 · p. 2
- Advanced tasks add constraints on speaking duration, rhythmic consistency or synchronization with a beat, and overlapping with the user in real time.These task families are Time, Tempo, and SimulSpeak, respectively.Source: §3.1 · pp. 2–3
3.2. Task Formalization
- Each instruction-following instance combines a base task with constraints that the model must satisfy while performing that task.The example of counting from one to ten in ten seconds combines sequence generation with range and duration constraints.Source: §3.2 · p. 3
3.3. Dataset Construction Pipeline
- The dataset pipeline creates seed instructions, diversifies them with LLM paraphrases and variable values, synthesizes speech, then filters transcripts and manually checks sample quality.For precise-tempo tasks, audio was manually edited; CosyVoice was the primary speech synthesis system.Source: §3.3 · p. 3
- The dataset contains 1,475 test instances: 700 Basic-task samples across 14 subtasks and 775 Advanced-task samples across 31 subtasks.Each Basic subtask has 50 samples and each Advanced subtask has 25.Source: §3.3 · p. 3
3.4. Dual-channel Evaluation
- The evaluation transcribes dual-channel user and model audio into time-aligned text for an LLM judge; the authors report that audio-LLM judging costs more and aligns less with human evaluation, while rule-based metrics can be too rigid.The authors favor text-based judging for reasoning about the core speech despite natural conversational preambles.Source: §3.4 · p. 3
- The text-based LLM judge is reported to align with human preferences and can also evaluate behaviors such as turn-taking.The paper describes it as a unified, simple, effective method; additional turn-taking results are on the project website.Source: §3.4 · p. 3
4. EXPERIMENTAL SETUP
The evaluation covers several full-duplex SLM designs and commercial voice agents, alongside an oracle topline. It uses LLM judging for timed transcripts and human evaluation on advanced tasks.
Models
- The evaluation compares time-multiplexing models Freeze-Omni and Unmute, dual-channel Moshi, and commercial agents Gemini-Live and GPT-realtime.The time-multiplexing systems use streaming encoders and decoders around a frozen LLM; Moshi directly processes and generates speech with a fine-tuned LLM.Source: §4 · p. 3
Oracle System
- SSML-LLM serves as an oracle topline, generating precisely timed responses from full word-level user alignments before TTS synthesis.It is non-streaming and non-causal, so it is not feasible for real-time use; it calibrates the LLM judge and human evaluations.Source: §4 · pp. 3–4
Instructions
- The task instructions prioritize natural language and time-related cues over commands made unequivocally precise.The authors state that overly constrained instructions would make conversations unnatural.Source: §4 · p. 3
Evaluation
- The setup includes LLM-judge scores on Basic and Advanced Tasks and human evaluation on Advanced Tasks.The human-evaluation figure covers Game-Time Advanced Tasks.Source: Figure 3 · p. 4, Figure 4 · p. 4
- The LLM-as-a-judge evaluation uses Whisper-medium for word-level alignments and Gemini 2.5 Pro to assess time-stamped transcripts.The authors selected Gemini 2.5 Pro after a preliminary study comparing its temporal-behavior evaluation with other open-source and commercial models.Source: §4 · p. 4
5. RESULTS
The results show that temporal constraints substantially challenge SLMs, even when models handle basic tasks. Human evaluations follow a similar trend to LLM judging, which correlates reasonably well with human scores.
5.1. Main Results
- The oracle leads across Basic Tasks, GPT-realtime performs strongly on most, and time-multiplexing models generally outperform dual-channel Moshi.GPT-realtime is the only model described as delivering reasonable performance on Repeat; the authors note room for improvement in academic models.Source: §5.1 · p. 4
- Temporal constraints cause a substantial performance drop; models do better on Time-Fast and Time-Slow than Time-Silence, while Tempo and SimulSpeak remain difficult even for GPT-realtime.The authors interpret the pattern as evidence that models can adjust speaking rate but struggle with precise temporal requirements.Source: §5.1 · p. 4
5.2. Human Evaluation
- Across 4 models × 35 data scores, dual-channel LLM judging correlates with human scores (Spearman’s ρ = 0.677; Pearson’s r = 0.675).Human evaluation shows a similar performance trend; the authors note LLM judging may be more objective for precise measurements such as ten-second silence.Source: §5.2 · p. 4, Table 3 · p. 4
6. CONCLUSION
The conclusion presents Game-Time as addressing a gap in temporal-dynamics evaluation for conversational SLMs. Its results show that nearly all models struggle when temporal constraints are introduced, while dual-channel LLM judging offers a reliable, scalable evaluation method.
6. CONCLUSION
- Game-Time evaluates SLM timing, tempo, and simultaneous speaking, addressing a gap in evaluation of conversational temporal dynamics.The evaluation uses a series of tasks testing these temporal capabilities.Source: §6 · p. 4
- Nearly all models fail under temporal constraints, revealing persistent difficulty managing precise timing or synchronizing with users, even in advanced systems.The authors report that dual-channel LLM reasoning provides a reliable, unified, and scalable way to measure these behaviors.Source: §6 · pp. 4–5