RoboChrono: A Real Robot Benchmark for Streaming Task Understanding

1 Introduction

• Robot-mounted occlusion, motion, and partial observability make single observations insufficient for inferring task state, completed actions, and what may happen next.The benchmark restricts models to observations available up to the queried moment, without subsequent frames.Source: §1 · pp. 1–2
• RoboChrono covers 39 manipulation scenarios, 1,816 successful execution videos, approximately 25 hours of data, and 34,713 evaluation instances.The collection combines teleoperated robot executions with bare-hand human demonstrations in five scenarios.Source: §1 · pp. 2–3
• Seven tasks assess action interpretation and anticipation, visual evidence association, and temporal organization under a streaming observation boundary.Figure 1 depicts the task distribution and limits each query to observations available up to that moment.Source: §1 · p. 2, Figure 1 · p. 2
• The paper evaluates 18 open-source and commercial multimodal models across all 39 scenarios in a zero-shot protocol.The authors report that capability differences, especially in temporal ordering, localization, and future-action reasoning, can be obscured by aggregate scores.Source: §1 · p. 2
• Table 1 compares benchmarks by data sources, robot-mounted views, collection and annotation protocols, and evaluation scale.RoboChrono uses real-robot executions, complementary onboard views, and human-defined temporal annotations under restricted observation histories.Source: Table 1 · p. 3, §1 · p. 3

Egocentric Video Understanding

• Egocentric benchmarks address temporal reasoning, task reasoning, and forecasting, but primarily use human-worn videos with viewpoints and motion patterns unlike robot-mounted cameras.Examples include Ego4D, Ego Exo4D, EgoTaskQA, EgoSchema, EgoPlan Bench, and VidEgoThink.Source: §2 · p. 3

Robot-Centric Embodied Understanding

• Robot-centric benchmarks connect visual reasoning with interaction and planning, but often use heterogeneous data sources, embodiments, camera configurations, or annotation pipelines.The surveyed work spans video question answering, robot trajectories, state transitions, embodied perception, spatial reasoning, affordances, and planning.Source: §2 · p. 3

Streaming Video Understanding

• Streaming video benchmarks evaluate timestamped or online reasoning, but mainly use general videos and human-centered activities rather than structured physical robot manipulation.StreamingBench and OVBench are cited as examples of online video evaluation.Source: §2 · p. 3

RoboChrono

• RoboChrono combines controlled real-robot executions, head and wrist views, human-annotated temporal events, and restricted observation histories in one benchmark.Its scenarios include diverse manipulation objects and action primitives across 39 real-world scenarios.Source: §2 · p. 3, Figure 2 · p. 4

3.1 Benchmark Overview

• RoboChrono evaluates seven tasks using observed video prefixes and task-specific queries, with multiple-choice answers or temporal intervals as outputs.Explicit task goals are supplied for goal-conditioned next-action prediction.Source: §3.1 · p. 4

3.2 Data Collection and Temporal Annotation

• The dataset covers 39 tasks—29 gripper, five dexterous-hand, and five ego-hand—with 1,816 successful videos, approximately 25 hours, and 34,713 evaluation instances.Each execution has action and state intervals that provide reference semantics and temporal boundaries for sampling evaluation points and constructing questions and answers.Source: §3.2 · p. 4

3.3 Evaluation Protocol

• At each evaluation time, models receive only visual observations available up to that moment; future visual evidence remains hidden at inference.Later annotations may determine next-action reference answers during construction, while matching, ordering, and localization instances also respect the permitted observation range.Source: §3.3 · pp. 4–5
• The zero-shot protocol standardizes task definitions and scoring; six multiple-choice tasks use accuracy, while localization uses Recall@1 at tIoU 0.5.Runs with response parse failure rates of 50% or higher are excluded, but valid low-scoring predictions remain in comparisons.Source: §3.3 · pp. 5–6

4.1 Experimental Setup

• The zero-shot comparison uses common task definitions and scoring rules, with no benchmark-specific supervised fine-tuning.Accuracy is reported for six multiple-choice tasks; Action Time Localization is scored separately using Recall@1 at tIoU 0.5.Source: §4.1 · p. 6

4.2 Main Results

• 95.4% Frame Matching but 32.9% Frame Ordering for RynnBrain1.1-122B-A10B illustrates that strong visual correspondence does not ensure temporal-order understanding.Goal conditioning consistently improves next-action prediction; GPT-6-Astra rises from 76.4% to 83.6%.Source: §4.2 · pp. 6–7
• 80.5% Choice Avg. makes GPT-6-Astra the strongest overall model; it leads all seven dimensions, including 98.3% Frame Matching and 68.3% Frame Ordering.Choice Avg. excludes Action Time Localization, which is reported separately as R@1 at tIoU ≥0.5.Source: §4.2 · p. 6, Table 3 · p. 7
• 33.2%–36.0% Choice Avg. scores for four lower-end models accompany broad weaknesses in temporal grounding and cross-observation reasoning.The four models are InternVL3-2B, Cosmos3-Edge-2B, SenseNova-SI-1.1-2B, and Qwen3-VL-2B.Source: §4.2 · p. 7

4.3 Effect of Robot Collection Platform

• 5.8 percentage points higher Choice Avg. on Tianji than GIM reflects a moderate platform sensitivity, while the capability pattern remains broadly similar.The comparison matches 12 tasks recorded on both platforms; the largest differences are about 9.6 points for Current Action Recognition and Frame Matching, versus 2.9 for Frame Ordering.Source: §4.3 · pp. 7–8, Table 4 · p. 8, Figure 5 · p. 9

4.4 Input Ablation

• −22.1 points for Current Action Recognition and −30.2 for Frame Matching follow removal of visual observations, while Next Action Prediction changes by less than one point.Using only the current frame instead of observation history reduces Current Action Recognition by 22.5 points but has only a small effect on future action prediction.Source: §4.4 · p. 8, Table 5 · p. 9
• Shuffling frames reduces Current Action Recognition by 8.9 points and Action Time Localization from 30.4 to 12.1, but increases Frame Matching.The identity re-encoding control produces almost no change, and the authors interpret the results as task-specific use of temporal order.Source: §4.4 · p. 9, Figure 5 · p. 9
• The ablations distinguish recognizing visible evidence from tracking task progress: temporal context matters for current-state and temporal grounding, while next-action prediction is less sensitive to visual history.The authors suggest next-action prediction relies more on task priors.Source: §4.4 · pp. 8–9

• RoboChrono evaluates temporal task understanding from progressively revealed visual observations across 39 manipulation scenarios and seven complementary tasks.The tasks cover action understanding, anticipation, cross-view and temporal association, ordering, and temporal localization.Source: §5 · p. 9
• The results reveal a gap between recognizing visible content and understanding how a physical task unfolds over time.The evaluation identifies weaknesses in temporal ordering, future action prediction, viewpoint alignment, and temporal localization.Source: §5 · p. 9
• The findings highlight evaluating temporal capabilities individually rather than relying solely on aggregate performance.The authors present RoboChrono as a testbed for developing systems that reason more reliably about task progress during ongoing real-world robot interaction.Source: §5 · p. 9

• The project website is https://continuity3.github.io/robochrono/.Source: Resources · p. 9
• The code repository is https://github.com/mfan-res/ ROBOCHRONO.Source: Resources · p. 9
• The Tianji and GIM dataset repositories are https://huggingface.co/datasets/gimai/RC-Tianji and https://huggingface.co/datasets/gimai/RC-GIM, respectively.Source: Resources · p. 9

• Generative AI tools were used only for language editing and writing assistance.Source: AI Use Statement · p. 10
• AI tools were not used for data generation, experimental execution, result analysis, or scientific decision making.Source: AI Use Statement · p. 10
• The authors reviewed all AI-assisted content and take full responsibility for the final manuscript.Source: AI Use Statement · p. 10

Hardware Platforms

• The benchmark uses a dual-arm whole-workspace system and two tabletop manipulators, both operated through VR-based teleoperation.The platforms differ in robot embodiment and workspace configuration.Source: A Supplementary Material · p. 12, Figure 6 · p. 12

Conditional Action Time Evaluation

• Conditional Action Time scoring evaluates parseable intervals expressed in seconds, excluding normalized answers from the denominator while keeping the reference interval and threshold unchanged.The conditional score addresses systems that return fractions of video duration despite prompts requiring seconds.Source: A Supplementary Material · p. 12

Qualitative Action Time Examples

• Figure 7 presents six GIM and Tianji localization examples from five models, showing accurate alignment and failures such as overly broad, non-overlapping, or out-of-video intervals.Panels (a)–(c) enlarge GIM examples and panels (d)–(f) enlarge Tianji examples.Source: A Supplementary Material · p. 13, Figure 7 · pp. 13–14

RoboChrono Evaluation Pipeline

• Algorithm 1 validates configurations, prepares visual input, runs inference, parses choices or intervals, scores accuracy or tIoU@0.5, and aggregates question results into a report.Its inputs are models, scenarios, and evaluation dimensions; its output is an evaluation report.Source: A Supplementary Material · p. 15