PaperlayerView exact revision on arXiv

Source-linked AI summary

ChronoGraph: Functional 4D Scene Graphs with Vision-Language Models for Interaction Understanding and Grounded Planning

Chenyangguang Zhang, Malgorzata Gwiazda, Guanlong Jiao, Yuanchen Ju, Federico Tombari, Koushil Sreenath, Marc Pollefeys, Sunghwan Hong

arXiv:2609.39665v1cs.AI

TL;DR

Embodied agents need to interpret how actions change a scene and use that state to plan grounded next actions, capabilities that existing benchmarks often assess separately. ChronoGraph provides a shared affordance-grounded representation, and ChronoGraphBench and two-stage-trained ChronoGraphVLM support both tasks. The models improve over pretrained baselines across scales, transfer zero-shot to VLM4D, and support real-world mobile manipulation through existing robot skills without additional fine-tuning.

  • Problem

    Existing VLM benchmarks largely assess spatial reasoning, temporal scene change, and planning separately, leaving their connection through affordance grounding and successor-state prediction underexplored.

    Source: §1 · p. 2

  • Method

    ChronoGraph represents affordance-level actions and semantic and geometric state changes in a shared graph, with ChronoGraphVLM trained to produce graph traces before answering and then optimized through joint graph reinforcement learning.

    Source: §1 · pp. 2–3

  • Results

    ChronoGraphVLM improves over corresponding pretrained baselines across model scales in both understanding and planning, with overall scores rising by 24.0 at 9B and 24.1 at 4B, and transfers zero-shot to VLM4D.

    Source: §1 · p. 2, §6.2 · p. 7

  • Takeaways & Limitations

    Real-world demonstrations show that graph-based planning and affordance grounding support sequential mobile manipulation through existing robot skills without additional fine-tuning.

    Source: §1 · p. 2, §7 · p. 10

Abstract

from arXiv · show

Embodied agents must determine where to act, anticipate the resulting scene changes, and interpret observed outcomes to guide subsequent actions. This requires connecting 4D interaction understanding, which explains how past actions changed the scene, with spatially grounded planning, which determines how and where to act toward a goal and anticipates the resulting scene changes. We introduce ChronoGraph, a functional 4D scene graph that links actions on affordance parts to semantic and geometric state changes. By representing observed and anticipated transitions in the same form, it provides a shared basis for understanding and planning. We construct ChronoGraphBench through an automatic data engine that converts human-interaction videos and simulated robot trajectories into graph-annotated questions for training and evaluating Vision-Language Models (VLMs) on both tasks. Using these annotations, we train ChronoGraphVLM by adapting pretrained VLMs in two stages. Graph-as-Chain-of-Thought supervised fine-tuning teaches the models to reconstruct observed transitions and predict future ones as graph traces before answering. Subsequent joint 4D graph reinforcement learning directly rewards graph properties and answer correctness. Experiments across model scales show improvements over the corresponding pretrained baselines and zero-shot transfer to VLM4D. Real-world demonstrations further show that graph-based planning and affordance grounding support mobile manipulation through existing robot skills without additional fine-tuning.

1 INTRODUCTION

Embodied interaction requires connecting interpretation of realized scene changes with planning grounded in affordances and anticipated outcomes. ChronoGraph supplies a shared representation for these tasks, and ChronoGraphBench and ChronoGraphVLM support training and evaluation across them.

1 INTRODUCTION

  • Existing VLM benchmarks generally assess spatial reasoning, temporal scene change, and planning separately, with limited attention to affordance grounding and successor-state prediction.This leaves the link between understanding an interaction and planning the next one underexplored, especially when actions change spatial configuration or affordance accessibility.Source: §1 · p. 2
  • ChronoGraph represents affordance-level actions and their semantic and geometric effects so observed interactions and anticipated outcomes share a common structure.That structure connects interpreting realized interactions with planning the next grounded action.Source: §1 · pp. 2–3
  • ChronoGraphVLM improves over pretrained baselines across model scales in interaction understanding and planning, transfers zero-shot to VLM4D, and supports real-world mobile manipulation through existing skills without additional fine-tuning.The real-world demonstrations use explicitly generated functional 4D scene graphs together with learned planning and affordance grounding.Source: §1 · pp. 2–3
  • ChronoGraphBench provides graph-based data for training and evaluating interleaved interaction-understanding and planning tasks.It is constructed from egocentric human-interaction videos and simulated robot trajectories.Source: §1 · pp. 2–3
  • ChronoGraphVLM is trained with Graph-as-Chain-of-Thought supervised fine-tuning followed by joint 4D graph reinforcement learning.The supervised stage teaches graph traces for observed and future transitions before answers; reinforcement learning gives feedback on graph properties and answer correctness.Source: §1 · pp. 2–3

2 RELATED WORK

Prior VLM work addresses spatial and 3D reasoning, temporal scene change, and embodied planning, but these directions largely treat them separately. ChronoGraph instead links observed interactions to future actions through functional affordance parts and their semantic and geometric changes.

3D and 4D Understanding with VLMs

  • 3D and 4D VLM work emphasizes spatial relations, object or camera motion, and observed dynamics, whereas this paper connects observed interactions to future actions at the level of functional affordance parts.The proposed task also requires localizing interactions, inferring semantic and spatial effects, and using the resulting state to ground the next action and predict its consequences.Source: §2 · p. 3

Embodied Planning with VLMs

  • Prior embodied-planning work includes graph representations of task-relevant entities, while ChronoGraph links affordance-level actions to semantic and geometric changes in a shared temporal graph.MomaGraph represents objects, interactive parts, and spatial and functional relations as plain text for static scenes.Source: §2 · p. 3

3 CHRONOGRAPH: FUNCTIONAL 4D SCENE GRAPH FOR UNDERSTANDING AND PLANNING

ChronoGraph extends functional scene graphs to represent action-triggered semantic and spatial changes. Its graph states and transitions capture affordance-grounded actions and the evolving scene needed for a goal.

3 CHRONOGRAPH: FUNCTIONAL 4D SCENE GRAPH FOR UNDERSTANDING AND PLANNING

  • ChronoGraph extends functional 3D scene graphs to model the action-triggered 4D evolution of an interactive scene.It represents objects and affordances at a timestep as well as semantic and spatial changes induced by interaction.Source: §3 · p. 3
  • Each graph state contains task-relevant object and affordance-part nodes with semantic and 3D attributes, linked by directed functional and spatial relations.The graph therefore records which entities and actionable parts are present and how their states relate.Source: §3 · p. 3
  • An interaction is grounded on an affordance node and updates relevant node states, geometric attributes, and relation edges.Each transition captures one action's local effect; chaining transitions represents evolving functional and spatial state toward the goal.Source: §3 · pp. 3–4

4 CHRONOGRAPHBENCH: DATASET AND BENCHMARK

ChronoGraphBench uses human and simulated-robot interactions to create graph-annotated data for structured reasoning and grounded VQA. Its annotation and question-generation processes support evaluation of observed transitions and goal-directed planning under different conditioning settings.

4 CHRONOGRAPHBENCH: DATASET AND BENCHMARK

  • The dataset combines egocentric videos from HD-EPIC and HOI! with synthetic robot trajectories from RoboCasa to annotate affordance-level actions and semantic and spatial changes.The graph annotations support structured reasoning supervision and generation of 4D-grounded VQA examples.Source: §4 · p. 4
  • ChronoGraphBench contains 1,669 interaction samples, 5,143 graph states, and 15,094 QA pairs across 357 object categories.The data come from human egocentric recordings and robot simulations, with questions spanning conditioning settings and reasoning categories.Source: §4 · p. 4

4.1 CHRONOGRAPH ANNOTATION

  • The shared annotation pipeline initializes semantic graphs, grounds objects and affordances in 3D, and uses human review and graph refinement when checks find incomplete, ambiguous, or low-confidence fields.Dataset metadata, annotations, or simulator ground truth substitute for pipeline components when available.Source: §4.1 · pp. 4–5

4.2 QUESTION CONSTRUCTION AND EVALUATION TASKS

  • Question generation uses 62 graph-derived templates for atomic, short-horizon, and long-horizon interactions, with graph constraints and plausible distractors validated through iterative review.For atomic action questions, candidates are rejected when video frames do not improve answer correctness over the same textual prompt across three repeated runs; this check is not applied to longer-horizon questions.Source: §4.2 · p. 5
  • The benchmark evaluates realized transitions, static-conditioned next-action and successor-state prediction, and history-conditioned planning from a goal and video prefix.Static-conditioned planning includes grounding the interaction location; history-conditioned planning uses completed sub-actions to infer the resulting state and predict what comes next. Questions cover action, affordance, state-change, spatial, and 2D/3D grounding reasoning.Source: §4.2 · p. 5

5 CHRONOGRAPHVLM: GRAPH-GUIDED LEARNING FOR UNDERSTANDING AND PLANNING

ChronoGraphVLM learns to generate functional 4D scene graph traces before answering, using supervised fine-tuning followed by joint graph reinforcement learning. The traces represent observed transitions for interaction understanding and future actions and states for planning.

5 CHRONOGRAPHVLM: GRAPH-GUIDED LEARNING FOR UNDERSTANDING AND PLANNING

  • ChronoGraphVLM first learns structured graph generation and graph-conditioned answering with supervised fine-tuning, then optimizes graph consistency and answer quality through joint reinforcement learning.Source: §5 · p. 5

5.1 GRAPH-AS-CHAIN-OF-THOUGHT SUPERVISED FINE-TUNING

  • Supervised fine-tuning trains an ordered graph–evidence–answer trace with autoregressive likelihood, linking the final answer to selected supporting graph properties.Source: §5.1 · pp. 5–6
  • Interaction understanding reconstructs observed graph transitions, while history- and static-conditioned planning predict future actions and resulting states toward a task goal.The output traces serialize geometric information as 3D centers in the input frame’s coordinate system, or the current-image frame for static-conditioned planning.Source: §5.1 · p. 6

5.2 JOINT 4D GRAPH REINFORCEMENT LEARNING

  • The graph reward compares aligned states and transitions, excluding transitions with negligible reference motion because their direction is unstable.Needleman–Wunsch alignment uses graph-state similarity; graph scoring covers node names, edge types, semantic states, state counts, changes, and action sequences.Source: §5.2 · pp. 6–7
  • Multiple-choice answers receive exact-match reward, while grounding reward decreases smoothly with distance from the target box and is full inside it.The composite reward coefficients weight format validity, graph quality, and answer correctness.Source: §5.2 · pp. 6–7

6 EXPERIMENTS

ChronoGraphVLM improves across interaction understanding and planning, transfers to VLM4D, and supports sequential mobile-manipulation demonstrations. The experiments also compare training choices and evaluate them under held-out household- or simulation-level splits.

6.1 EXPERIMENTAL SETUP

  • Household- or simulation-level splits allocate 20%/68%/12% to supervised fine-tuning, reinforcement learning, and holdout; the evaluation set contains 1,811 VQAs from 202 interactions.Comparisons include commercial, Qwen-family, and specialized baselines; videos are uniformly sampled at 4 FPS for interaction understanding and history-conditioned planning.Source: §6.1 · p. 7

6.3 ZERO-SHOT CROSS-BENCHMARK RESULTS

  • On VLM4D, ChronoGraphVLM achieves 67.1% accuracy at 9B and 61.0% at 4B, improving over pretrained initializations by 6.6% and 7.3%, respectively.Evaluation is zero-shot on VLM4D's egocentric split.Source: Table 2 · p. 7, §6.3 · p. 8

6.2 MAIN RESULTS

  • ChronoGraphVLM raises overall score by 24.0 at 9B and 24.1 at 4B, reaching 75.2 and 73.0 and surpassing evaluated open-source and specialized baselines.Table 1 covers 4D interaction understanding and history- and static-conditioned planning.Source: §6.2 · p. 7, Table 1 · p. 8

6.4 ABLATION STUDIES

  • Training-free Graph-as-CoT prompting improves ChronoGraphBench overall by 7.8 over direct answering and 3.2 over native free-form thinking.It also improves zero-shot VLM4D accuracy by 3.4% and 1.7%, respectively; graph-guided models do not produce longer outputs on average than free-form reasoning.Source: §6.4 · pp. 8–9
  • Graph-as-CoT supervision reaches a ChronoGraphBench overall average of 69.9 and VLM4D accuracy of 64.2%, while answer-only supervision reaches 67.0 and 49.4%.Answer-only supervision raises the ChronoGraphBench average from the 51.2 baseline but reduces VLM4D accuracy from the 60.5% baseline.Source: §6.4 · p. 9
  • With answer-only reward, ChronoGraphBench overall score rises from 69.9 to 73.0 and VLM4D accuracy from 64.2% to 65.4%; adding graph rewards raises them further to 75.2 and 67.1%.Source: §6.4 · p. 9

6.5 APPLICATION: MOBILE MANIPULATION

  • Spot demonstrations span switch activation, cabinet opening with toy retrieval, and a longer kettle-retrieval and stovetop-placement task using existing robot skills.Before each subaction, the model uses observation history to predict actions and ground the target affordance in the current image.Source: §6.5 · p. 9

7 CONCLUSION AND LIMITATIONS

ChronoGraph links affordance-level actions to semantic and geometric scene changes, and the resulting models improve over pretrained baselines, transfer to VLM4D, and support real-world mobile manipulation. The paper identifies annotation cost and inference latency as limitations.

Contributions

  • ChronoGraph connects interaction understanding and spatially grounded planning through affordance-level actions linked to semantic and geometric changes.ChronoGraphBench and two-stage ChronoGraphVLM training support the approach; experiments report gains across scales, zero-shot VLM4D transfer, and mobile manipulation without additional fine-tuning.Source: §7 · p. 10

Limitations

  • Affordance-level 3D annotation cost limits dataset scale, while inference latency hinders real-time closed-loop replanning.The paper proposes broader environment and robot coverage, distillation, and streaming inference as future directions.Source: §7 · p. 10

Conclusion

  • The paper identifies learning how interactions change scenes and inform subsequent actions as a value of the approach.Source: §7 · p. 10

Related papers

Loading 2609.39665v1…