PaperlayerView exact revision on arXiv

Source-linked AI summary

RGraph-Eval: Task-Grounded Scene Graphs for Evaluating Robot Video Generation

Aleksandr Lazarev, Sofia Zimina, Roman Kuznetsov, Tatiana Zemskova, Iaroslav Koshelev, Dmitry Yudin

arXiv:2610.31121v1cs.RO

TL;DR

Robot videos can appear realistic while depicting incorrect object motion, interactions, or task outcomes, making visual appearance alone an incomplete basis for assessment. RGraph-Eval compares reference and generated videos through task-grounded temporal scene graphs and scores visible objects, relation changes, and 3D path shape. Across the video-generation benchmark, it shows strong agreement with human judgments, with its components contributing differently to task-completion and overall human alignment.

  • Problem

    Generated robot videos can look convincing while moving the wrong object, losing a grasp, or failing to place an object at its destination.

    Source: §1 · p. 1

  • Method

    RGraph-Eval represents task-relevant entities, interactions, and motion in temporal scene graphs and compares graphs from generated videos and reference demonstrations.

    Source: Prediction · p. 3

  • Results

    Across the video-generation benchmark, RGraph-Eval shows strong agreement with human judgments, and its components contribute differently to task-completion and overall human alignment.

    Source: §5 · p. 11

  • Takeaways & Limitations

    The framework provides interpretable evaluation based on task entities and interactions, while its graph annotations are introduced as reusable beyond evaluation.

    Source: §5 · p. 11

    Extraction errors and single-reference comparison remain limitations; valid alternative executions may receive lower scores, and the representation focuses on one manipulated object without certifying executable actions.

    Source: §5 · p. 11

Abstract

from arXiv · show

Generated robot videos can look convincing while depicting the wrong interac- tion: a different object is moved, a grasp disappears, or a placement ends out- side its destination. We introduce RGraph-Eval, a reference-based framework that makes these discrepancies explicit through task-grounded temporal scene graphs. Unlike approaches that construct separate evidence representations for different evaluation dimensions, we organize task entities, interactions, and motion within a unified, interpretable scene graph representation. A foundation-model pipeline identifies manipulation roles, grounds persistent object masks, infers and verifies state transitions and action intervals, and reconstructs manipulated- object motion in three dimensions. Applying this representation to a generated video and a reference demonstration supports an interpretable RGraph-Eval score with visible-object, relation-change, and 3D path-shape components. We also introduce ShareRobot-SG, a collection of 1,000 manipulation scenes from ten ShareRobot sources annotated with the multimodal representation. Our eval- uation separates graph-generation quality from agreement with human video judgments, and includes comparisons with adapted SVG2 and SG-Ego, a bench- mark of three video generators. The framework connects robot-video assessment to the observable entities and interactions underlying a task, while making the graph annotations reusable beyond evaluation. Code will be publicly available at https://github.com/SashaDance/rgraph-eval.

1 Introduction

Generated robot videos may be useful for inspection and data generation, but visual realism alone does not establish that they depict the intended manipulation. A plausible video can move the wrong object, omit a grasp, or fail to achieve the target state.

1 Introduction

  • Generated videos may support behavior inspection, visual demonstrations, and robot-learning data generation, but their quality must be assessed beyond visual appearance.Evaluation should check whether behavior preserves task-relevant entities, interactions, and final states.Source: §1 · p. 1
  • A plausible plate-stacking video may move the wrong plate, omit the grasp, or leave the white plate beside the red one.Source: §1 · p. 1

1 Conditioning First frame Task

The supplied passage gives the instruction to place a blue cup inside a brown cup.

1 Conditioning First frame Task

  • The example instruction is to place a blue cup inside a brown cup.Source: §1 · p. 2

2 Graph Generation based on Foundation Models

The supplied passage names segmentation and object masks.

2 Graph Generation based on Foundation Models

  • The passage identifies segmentation and object masks.Source: §2 · p. 2

Prediction

RGraph-Eval uses task-grounded temporal scene graphs to connect video assessment to visible task entities and interactions. Its foundation-model pipeline generates grounded graphs, while visual, relation, and motion comparisons support evaluation.

Prediction

  • Visible-object, relation-change, and 3D path-shape comparisons contribute to the RGraph-Eval score.Source: Figure 1 · p. 2
  • Existing evaluators assess complementary qualities, but their scores need not identify which visible object or interaction explains a disagreement.Source: Prediction · pp. 2–3
  • Task-grounded temporal scene graphs connect assessment to observable task evidence by representing goal-relevant entities, image regions, changing interactions, and motion.Source: Prediction · p. 3
  • Incorrect object assignments can propagate through tracking, relations, and geometry, while language models may infer goal-consistent rather than observed outcomes.The framework addresses these risks with visual grounding, persistent tracking, and independent verification, and evaluates graph quality separately from human alignment.Source: Prediction · p. 3
  • The framework introduces a foundation-model pipeline for grounded temporal graphs, ShareRobot-SG with 1,000 scenes from ten manipulation sources, and reference-based RGraph-Eval assessment.The assessment has an explicit component decomposition and is evaluated through human alignment, generator comparison, and component analysis.Source: Prediction · p. 3

2 Related Work

Existing video evaluators assess appearance, consistency, instruction following, physical plausibility, or preference, but may not identify which visible object or interaction explains a disagreement. This work positions task-grounded graphs as a manipulation-focused representation of object identity, interactions, and motion, while noting related graph extraction and robotics datasets.

2.1 Evaluating Generated Videos

  • Frame-wise similarity and broad video benchmarks do not explicitly establish manipulated-object identity, coherent interaction, or task completion.Human-aligned and reference-based methods assess additional dimensions, while this work uses grounded objects, masks, relation changes, and geometry as the comparison interface.Source: §2.1 · p. 3

2.2 Scene Graph Generation from Video

  • SVG2 and SG-Ego provide broad-vocabulary video graph comparisons, but geometric scene relations alone do not determine whether manipulation was executed correctly.The pipeline instead combines open-vocabulary object descriptions with task roles and an explicit state–action schema.Source: §2.2 · p. 3

2.3 Scene Graphs in Robotics and Annotation

  • Robotics uses scene graphs for plan verification, skill composition, and hierarchical action planning, while manipulation datasets with explicit graph annotations remain relatively scarce.ShareRobot-SG augments ShareRobot videos with object identities, masks, changing relations, action timings, and estimated 3D motion.Source: §2.3 · p. 4

3 Method

The method independently converts reference and generated videos, conditioned on the task goal, into temporal graphs that combine grounded entities, tracked masks, verified interactions, and estimated manipulated-object motion. RGraph-Eval compares those graphs using visible-object agreement, relation-event agreement, and 3D path shape.

3 Method

  • The same extractor processes the task goal, reference demonstration, and generated video independently, although the reference shows only one valid execution.The pipeline takes the goal and ordered RGB frames, selects grounding keyframes internally, and requires no external masks, hand-labelled boxes, or demonstration-planning steps.Source: §3 · p. 4

3.1 Role Grounding

  • The pipeline grounds the robot, manipulated object, initial support, and target, with an optional whole-parent entity for part–whole interactions.Descriptions and boxes identify entities, and visual evidence takes precedence over the goal when they conflict.Source: §3.1 · p. 5

3.2 Mask Tracking

  • SAM 3 generates candidate mask tracks from text descriptions, and overlap with grounding boxes selects intended tracks when objects look similar.SAM 2.1 propagates masks only where the selected SAM 3 track has gaps, preserving existing SAM 3 masks; stored masks and image-space measurements link entities to the video.Source: §3.2 · p. 5

3.3 Relations and Actions

  • Relations encode interaction states, while actions mark events over time, such as holding versus grab and release.A verification pass checks object identity, contact and lift timing, and final placement against frames and mask overlays, removing unsupported facts rather than assuming the goal was achieved.Source: §3.3 · p. 5

3.4 3D Reconstruction

  • Depth, camera intrinsics, and camera-to-world transforms support a manipulated-object 3D position estimate from valid-depth pixels selected by its mask.The component-wise median tracks visible object surface, not full pose or center of mass; 3D comparison requires compatible coordinates and normalization.Source: §3.4 · pp. 5–6

3.5 Scene Graph Sequence

  • Each prepared frame's graph combines tracked entities and masks, verified relations and action intervals, and estimated 3D positions.Entity IDs remain fixed within each video, and visual evidence takes precedence over the goal when they conflict.Source: §3.5 · p. 6

3.6 Reference-Based RGraph-Eval Score

  • RGraph-Eval compares videos at up to 30 progress-aligned samples using visible-object agreement, ordered relation-event agreement, and manipulated-object 3D path shape.The path term penalizes missing observations and compares motion shape independently of absolute position and orientation; it does not verify the final destination.Source: §3.6 · pp. 6–7

4 Experiments

Experiments separately assess graph-generation and tracking quality on ShareRobot-SG and agreement with human judgments on a 30-scene video-generation subset. Across these evaluations, RGraph-Eval captures manipulation relations and complementary evidence about object identity and motion.

4.1 Datasets

  • ShareRobot-SG contains 1,000 scenes from ten source datasets, with 900 automatically annotated and 100 additionally manually annotated; a separate 30-scene subset supplies reference episodes.The manual annotations support graph-quality evaluation, while the 30-scene subset is used for generator comparisons and human-judgment alignment.Source: §4.1 · p. 7, §4 · p. 7

4.2 Comparison of Scene Graph Generation

  • Graph comparisons are restricted to robot, manipulated-object, initial-support, and target roles, and to holding/touching, on/supporting, inside, and attached_to relations or their synonyms.Other relation types are excluded for all compared methods.Source: §4.2 · pp. 7–8
  • The method provides stronger manipulation-relation extraction while maintaining competitive object-role grounding, with persistent masks linking entities across frames.The comparison evaluates object roles and a common manipulation-relation scope; persistent grounding also supports the trajectories used by the score.Source: §4.2 · pp. 7–9

4.3 Video Generation Evaluation

  • RGraph-Eval and RBench tie for the highest reported task-completion correlation; RGraph-Eval leads for visual quality and overall human rating, while Ref4D’s event score leads for object consistency.Correlations are computed across three generator means from 90 generated videos rated by one assessor.Source: §4.3 · p. 9, Table 3 · p. 9
  • LingBot-VA leads on manipulation relations and 3D motion, Cosmos 3 Nano on visible-object agreement, and WoW-1 also captures object-motion structure despite lower overall agreement.The comparison uses the same 30-scene generation subset.Source: §4.3 · p. 9, Table 4 · p. 10

4.4 Ablation Studies

  • Removing SAM2.1 missing-frame repair substantially reduces tracking recall; semantic prompting and temporal repair provide complementary benefits for persistent object grounding.The ablation compares four variants on 100 annotated videos using the same ten frames per video and fixed Qwen proposals.Source: §4.4 · p. 10, Table 5 · p. 11
  • Relation changes align most strongly as a standalone component with task completion; relations plus 3D path shape align most strongly with the overall human rating.Visible-object and 3D-motion components provide complementary information, and combining components generally improves alignment.Source: §4.4 · p. 10, Table 6 · p. 11

5 Conclusion and Future work

RGraph-Eval agrees strongly with human judgments while capturing complementary aspects of generated behavior, and its component signals contribute differently to task-completion and overall alignment. The conclusion also identifies extraction and reference limitations and outlines broader future evaluation directions.

5 Conclusion and Future work

  • RGraph-Eval evaluates robot videos through task-grounded scene graphs combining object grounding, interaction changes, and 3D motion.The framework is presented as an interpretable evaluation based on entities and interactions underlying the task.Source: §5 · p. 11
  • ShareRobot-SG provides 1,000 scenes with multimodal scene-graph annotations.Future work proposes broader evaluation across more video-generation models and a larger benchmark.Source: §5 · p. 11
  • RGraph-Eval shows strong agreement with human judgments and captures complementary aspects of generated behavior through visible objects, relation changes, and object motion.Its component signals contribute differently to task-completion and overall human alignment.Source: §5 · p. 11
  • Extraction errors and single-reference comparison remain limitations, and valid alternative executions may receive lower scores.Future directions include multi-object and bimanual representations, multi-reference comparison, and uncertainty-aware scoring.Source: §5 · p. 11

Related papers

Loading 2610.31121v1…