Realtime-Bench: Can Frontier Models Solve Real-time Interactive Problems?

1 INTRODUCTION

• Existing real-time evaluations emphasize streaming perception, conversation, and interruption handling, with limited coverage of closed-loop tasks where delayed actions can change outcomes.In these settings, actions affect subsequent environment states.Source: §1 · p. 2
• Realtime-Bench comprises 50 interactive game and web tasks where evolving states and limited opportunities make success depend on both action choice and timing.The paper frames real-time streaming capability through temporal state awareness, adaptive deliberation, and time-aware action selection.Source: §1 · p. 2
• Under Wall Clock, the best success rate is 44.0%, while the best Passˆ3 is below 37%, showing that real-time interaction remains challenging.Wall Clock advances the environment according to actual inference and serving time.Source: §1 · p. 2
• Wall Clock uses actual inference latency, Paused Clock removes inference delays, and Token Clock advances simulated time according to generated tokens before action commitment.Token Clock tests temporal awareness independently of physical serving speed.Source: Figure 2 · p. 2, §1 · p. 2
• Substantial performance gaps relative to Paused Clock persist under Token Clock, while higher reasoning effort improves success under Paused Clock but has mixed effects under temporal pressure.The findings reflect a trade-off between decision quality and timely execution.Source: §1 · p. 3

2.1 STREAMING MULTIMODAL MODELS AND BENCHMARKS

• Streaming-model benchmarks commonly assess perception, response timing, interruption handling, and conversational quality, leaving untested whether actions affect closed-loop outcomes.Realtime-Bench evaluates closed-loop interaction in web environments and games, where success depends on what actions models take and when.Source: §2.1 · p. 3

2.2 REAL-TIME EVALUATION IN INTERACTIVE ENVIRONMENTS

• Earlier interactive evaluations often pause environments during inference, separating task abilities from response speed; real-time variants let environments evolve but couple results to wall-clock latency.The cited real-time evaluations include games and web environments.Source: §2.2 · p. 3
• Token Clock maps generated tokens to environment time at a shared rate, preserving deliberation’s temporal cost while decoupling evaluation from physical serving speed.The passage relates this approach to Real-Time Reasoning Gym.Source: §2.2 · p. 3

3.1 REAL-TIME STREAMING CAPABILITIES

• Real-time streaming capability is effective perception, reasoning, and action while the environment evolves, organized as temporal state awareness, adaptive deliberation, and time-aware action selection.The dimensions cover tracking changes and information freshness, adjusting reasoning to available time, and choosing actions that remain useful when executed.Source: §3.1 · pp. 3–4
• In the ticket example, an agent tracks sale status, limits deliberation as the purchase window closes, and chooses whether to wait or buy while tickets remain available.The example connects the three capability dimensions to one evolving task.Source: §3.1 · p. 4

3.2 CLOCK PROTOCOLS

• Wall Clock advances the environment during inference according to actual elapsed time, so action timing depends on model computation and serving infrastructure.This protocol evaluates performance under real-world inference latency.Source: §3.2 · p. 4
• Paused Clock freezes the environment during inference, providing an idealized reference for comparing performance with and without inference delays in environment time.The protocol assesses task performance without inference’s temporal cost.Source: §3.2 · p. 4
• Token Clock advances the environment at a fixed rate per generation token, so more tokens delay action regardless of actual inference latency.The action is applied at the simulated time reached when it is committed.Source: §3.2 · p. 4
• Token Clock defines κ as the token-to-time rate using token budget B per observation interval and observation rate f; generating n tokens before commitment determines simulated delay.The passage gives κ = n Bf.Source: §3.2 · p. 4
• When generation spans observation intervals, Token Clock and Wall Clock skip intermediate frames and provide the most recent scheduled frame at the resulting simulated time.Both protocols require balancing additional deliberation against delayed action, which can let relevant events or opportunities pass.Source: §3.2 · p. 4

4.1 DESIGN PRINCIPLES.

• The design criteria call for realistic application and game interfaces, environments that evolve independently of model actions, and success dependent on both action choice and timing.Examples include changing seat availability, moving obstacles, limited action windows, and transient opportunities.Source: §4.1 · pp. 4–5

4.2 TASK CONSTRUCTION.

• Task construction adapts realistic web scenarios and popular open-source games, adjusting goals and temporal constraints so success requires timely action while remaining feasible for humans.The resulting benchmark contains 30 web tasks and 20 game tasks, with task-specific success criteria.Source: §4.2 · p. 5

4.3 TASK EXAMPLES AND DISTRIBUTION.

• The benchmark includes a live-auction task requiring a winning bid before the window closes and a snake game requiring the agent to eat five apples; Figure 3 shows scenario distribution.Figure 4 presents the auction and snake examples, including their time-critical properties and token-budget action windows.Source: Figure 3 · p. 4, §4.3 · p. 5, Figure 4 · p. 5

5.1 OBSERVATION AND ACTION INTERFACE

• At each scheduled observation, models receive a screenshot and retained interaction history, then generate executable GUI actions from a common action space with read-only Set-of-Marks grounding.The action space includes click, type, scroll, and wait.Source: §5.1 · p. 5

5.2 CONTEXT MANAGEMENT

• History is removed in blocks rather than every step, keeping the prefix unchanged between folds so models can reuse KV cache across steps.All models follow the same retention policy to reduce repeated prefill computation.Source: §5.2 · p. 6

5.3 EPISODE EXECUTION

• Action time equals observation time plus the clock-specific delay; episodes end at a task condition or benchmark horizon and are scored against task-specific success criteria.Wall Clock uses actual inference latency, Paused Clock sets the delay to zero, and Token Clock bases it on generation tokens; GUI execution latency does not advance environment time.Source: §5.3 · p. 6

6.1 EXPERIMENTAL SETUP

• The study evaluates 12 frontier models and compares three clock protocols at low and high reasoning effort on six representative models.The six-model comparison excludes decision models because they do not report output token counts needed for Token Clock; observations are set to 1 FPS.Source: §6.1 · p. 6, Figure 5 · p. 7

6.2 RESULTS

• 36% is the highest Pass^3 under Wall Clock, achieved by GPT-6 Astra (high), while most models incur latencies exceeding one second.Jev-1.13.0 has sub-second latency but Pass^3 of 8%.Source: §6.2 · p. 7
• 70.0% in PH falls to 14.0% in WH for Claude Opus 5 and to 32.0% for Gemini 3.8 Flash, with substantial gaps from Paused Clock also persisting under Token Clock.In TH, Claude Opus 5 reaches 30.0% and Gemini 3.8 Flash reaches 34.0%. Figure 5 compares the three timing protocols.Source: §6.2 · p. 7, Figure 5 · p. 7
• Higher reasoning effort improves success for all six models in Paused Clock, but its effects are mixed under Token Clock and Wall Clock.Examples include Claude Opus 5 falling from 44.0% in TL to 30.0% in TH, while Kimi-K3 rises from 24.0% to 36.0%; GPT-5.6 Sol falls from 16.0% in WL to 8.0% in WH, while Gemini 3.8 Flash rises from 24.0% to 32.0%. Further reasoning may improve action selection but reduce the opportunity to execute successfully.Source: §6.2 · pp. 7–8
• With low reasoning effort, relaxing temporal pressure generally improves success but is not monotonic for every model and leaves many tasks unsolved.Claude Opus 5 rises from 30.0% at B = 32 to 62.0% with an unlimited budget, and Kimi-K3 from 22.0% to 60.0%. Figure 6 shows the effect of temporal pressure.Source: Figure 6 · p. 7, §6.2 · p. 8

6.3 QUALITATIVE ANALYSIS

• Three recurring failure modes are inadequate urgency-aware deliberation, failure to update decisions as states change, and misjudging whether to act or wait.Examples include generating 1,845 tokens with 10 seconds left for a Gomoku move, continuing to move toward a wall despite a nearby apple, and waiting after a road begins collapsing.Source: §6.3 · p. 8

7.1 ADAPTIVE REASONING WITH TEMPORAL AWARENESS

• Adaptive reasoning must balance the potential decision benefit of further deliberation against the risk of leaving too little time to execute.The appropriate effort can change within a task as the environment changes.Source: §7.1 · p. 8

7.2 CONTINUOUS PERCEPTION AND ACTION

• At the experiments’ 1 FPS observation rate, brief events can occur between observations, even when inference adds no temporal cost.Higher observation rates could provide more timely feedback, but require low-latency processing and action revision.Source: §7.2 · p. 9
• Higher temporal resolution couples inference cost from processing more frames with the risk that compressed visual inputs discard brief, action-relevant changes.The discussion points to processing efficiency and preserving action-relevant information as paired design directions.Source: §7.2 · p. 9
• At higher frame rates, context can exceed budgets and ongoing inference can become stale as new observations arrive.The discussion calls for history management that preserves temporal information and backlog strategies that decide which frames to retain and whether to discard or revise stale plans.Source: §7.2 · pp. 9–10

8 CONCLUSION

• 44.0% was the best Wall Clock success rate, while the best Passˆ3 was 36%, across the evaluation of 12 frontier models.The conclusion reports substantial gaps relative to Paused Clock that persist under Token Clock.Source: §8 · p. 10
• Higher reasoning effort improves success under Paused Clock but has mixed effects under temporal pressure.Relaxing generation’s temporal cost generally improves performance but does not ensure success.Source: §8 · p. 10
• Realtime-Bench comprises diverse dynamic game and web tasks for evaluating real-time interactive capability in multimodal agents.The evaluation covers 12 frontier models and compares three clock protocols, including Token Clock, which preserves deliberation’s temporal cost independently of physical serving speed.Source: §8 · p. 10
• The findings motivate adaptive reasoning with temporal awareness and continuous perception and action so agents can coordinate deliberation and execution with available time.The benchmark provides a controlled setting for studying these capabilities and measuring progress toward effective real-time interaction.Source: §8 · p. 10

A.1 OBSERVATIONS AND ACTIONS

• Agents receive resized screenshots and interactable-element descriptions, then produce GUI actions using a shared action schema.Screenshots are resized to a maximum dimension of 768 pixels; the schema allows action durations from 0 to 10 seconds and key.repeat from 1 to 100.Source: §A.1 · p. 16, Table 1 · p. 16

A.2 CLOCK EXECUTION AND TOKEN ACCOUNTING

• Under Token Clock, the harness advances the environment to the action timestamp determined by observation time and charged output-token count.Physical inference and GUI execution latency do not advance simulated time; charged API tokens include reasoning and output tokens, with generated audio excluded for Gemini Live.Source: §A.2 · p. 16

A.3 CONTEXT MANAGEMENT

• Context management retains visual and textual history separately, replacing older screenshots or complete turns with placeholders or a compact action ledger when limits are reached.Retained messages preserve a stable context prefix for cache reuse where supported.Source: §A.3 · pp. 16–17

A.4 MODEL CONFIGURATION

• The evaluated model groups include asynchronous, streaming, and decision models, with decision models evaluated only under Wall Clock.Jev 1.13 is evaluated only on web tasks because its structured-text observations do not adequately capture game visual state; configurable reasoning effort uses provider-exposed low and high settings.Source: §A.4 · p. 17

• Table 2 reports per-run success rates and average decision latency.Source: Table 2 · p. 18
• Table 3 reports Passˆk, the rate of success on all k attempts averaged over combinations of three runs.Source: Table 3 · p. 19
• Table 4 reports Pass@k, the rate of success on at least one of k attempts averaged over combinations of three runs.Source: Table 4 · p. 19

B.2 TOKEN-RATE ABLATION

• Table 5 reports overall success rates across all 50 tasks under low reasoning effort; its B = ∞ condition removes simulated inference delay.The caption supplies the table’s scope but no cell values.Source: Table 5 · p. 20

C.1 SCOPE AND PROCEDURE.

• The first author developed an annotation codebook from failed trajectories; Codex applied it, targeted video frames verified candidate events, and the second author checked annotations.Frame review was targeted rather than exhaustive.Source: §C.1 · p. 20

ANNOTATION SCOPE

• RT-Deliberate is assessed only under token-based timing, while RT-Perceive and RT-Act are assessed only in PH; out-of-setting pairs are out of scope, not negative judgments.RT-Deliberate concerns generation latency mismatched to the response window; RT-Perceive concerns behavior based on stale state; RT-Act concerns acting when waiting is appropriate or waiting when intervention is needed.Source: ANNOTATION SCOPE · pp. 20–21

EVIDENCE REQUIREMENTS

• Candidate events are assessed against observations, behavior, consequences, and alternative explanations, then classified as confirmed, probable, or insufficient evidence.Final outcomes may corroborate consequences, but hidden evaluation state alone cannot establish that an observable change was ignored.Source: EVIDENCE REQUIREMENTS · p. 21
• The main figure counts confirmed and probable assignments, pools episode counts using setting-specific denominators, and treats categories as evidence-supported attributions rather than an exhaustive causal decomposition.RT-Perceive and RT-Act may overlap in PH, so percentages need not sum to 100%; “Other” includes valid failed episodes without selected-category assignments at the reported threshold.Source: EVIDENCE REQUIREMENTS · p. 21

C.3 FAILURE EXAMPLES

• Other failures include continued leftward movement despite an apple’s location, waiting through a final save deadline, and deliberation that outlasts a 10-second guess clock.The snake hits the wall at 17 seconds; the shared-document deadline passes with the required date unchanged.Source: §C.3 · p. 21, Figure 10 · p. 22, Figure 11 · p. 22
• In timed examples, models deliberate past action windows, wait through actionable opportunities, or act after a price limit has been crossed.The examples include a 544-token bid decision with 1 second remaining, a 42.3-second delay before returning to a waitlist, and a purchase after BTC rose above $60,000.Source: §C.3 · p. 21, Figure 9 · p. 22, Figure 12 · p. 23
• In additional Wall Clock runs, a model waits while a held room is ready to book, and another buys BTC after its price rises above the $60,000 limit.The examples also include a flight-assignment call that takes 16.6 seconds and returns wait despite the sequencing cutoff.Source: §C.3 · p. 22, Figure 14 · p. 23, Figure 15 · p. 24

D.1 PROMPT TEMPLATES.

• Each episode’s shared instructions are combined with domain, clock, and task instructions; the domain component identifies the setting and interaction requirements.The domain and clock instruction specifications are given in Appendices D.1.1 and D.1.2.Source: §D.1 · p. 25

D.1.1 DOMAIN INSTRUCTIONS.

• Game and web tasks use separate domain prompts.The prompts specify the interaction setting and behavioral constraints.Source: §D.1.1 · p. 25

D.1.2 CLOCK INSTRUCTIONS.

• The clocks differ in whether inference delay advances the environment: Token Clock charges output tokens, Paused Clock does not, and Wall Clock follows elapsed physical time.Under Token Clock, actions land after n/kappa simulated seconds; Paused Clock applies each action at the observed tick before advancing one frame interval.Source: §D.1.2 · pp. 25–26
• The action contract requires one valid executable action in the active action space, with canonical JSON preferred and all generated output tokens charged.The instructions define GUI action forms, screenshot-specific target marks, and handling for held inputs and invalid or multiple actions.Source: §D.1.2 · pp. 25–26