Source-linked AI summary
MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planning
Yuanchen Ju, Yongyuan Liang, Yen-Jen Wang, Nandiraju Gireesh, Yuanliang Ju, Seungjae Lee, Qiao Gu, Elvis Hsieh, Furong Huang, Koushil Sreenath
TL;DR
Household mobile manipulators need scene representations that capture spatial layout, object function, and actionable parts, but prior scene graphs often separate relations and miss task relevance. MomaGraph combines these elements and supports them with a dataset, benchmark, and reinforcement-learned VLM for Graph-then-Plan. The model outperforms open-source baselines and is reported to generalize across benchmarks and real-robot experiments.
Problem
Existing scene graphs often separate spatial and functional relations and lack dynamic, task-relevant information needed to represent household scenes for embodied agents.
Source: §1 · p. 2Method
MomaGraph unifies spatial and functional relations with part-level interactive nodes, supported by MomaGraph-Scenes and MomaGraph-Bench, while MomaGraph-R1 generates task-oriented graphs and plans from them.
Source: §1 · pp. 2–3Results
MomaGraph-R1 surpasses open-source baseline models, with gains across public benchmarks and reported effectiveness in real-world robotic experiments.
Source: §1 · p. 3Takeaways & Limitations
The work provides a task-oriented scene representation and evaluation suite for studying embodied planning from fine-grained scene understanding through high-level task planning.
Source: §1 · pp. 2–3In real-robot evaluation, failures include graph-generation errors such as incorrect spatial relations or missing nodes, and suboptimal action ordering.
Source: §6.4 · p. 11
Abstract
from arXiv · showhide
Mobile manipulators in households must both navigate and manipulate. This requires a compact, semantically rich scene representation that captures where objects are, how they function, and which parts are actionable. Scene graphs are a natural choice, yet prior work often separates spatial and functional relations, treats scenes as static snapshots without object states or temporal updates, and overlooks information most relevant for accomplishing the current task. To address these limitations, we introduce MomaGraph, a unified scene representation for embodied agents that integrates spatial-functional relationships and part-level interactive elements. However, advancing such a representation requires both suitable data and rigorous evaluation, which have been largely missing. We thus contribute MomaGraph-Scenes, the first large-scale dataset of richly annotated, task-driven scene graphs in household environments, along with MomaGraph-Bench, a systematic evaluation suite spanning six reasoning capabilities from high-level planning to fine-grained scene understanding. Built upon this foundation, we further develop MomaGraph-R1, a 7B vision-language model trained with reinforcement learning on MomaGraph-Scenes. MomaGraph-R1 predicts task-oriented scene graphs and serves as a zero-shot task planner under a Graph-then-Plan framework. Extensive experiments demonstrate that our model achieves state-of-the-art results among open-source models, reaching 71.6% accuracy on the benchmark (+11.4% over the best baseline), while generalizing across public benchmarks and transferring effectively to real-robot experiments.
1 INTRODUCTION
MomaGraph addresses scene graphs that omit functional, dynamic, or task-relevant information with a unified representation, a dataset and benchmark, and a vision-language model for Graph-then-Plan task planning.
1 INTRODUCTION
- Existing scene graphs often separate spatial from functional relations, represent static scenes, and omit task-relevant information.The paper argues that an embodied-agent scene graph should integrate spatial and functional relations, include fine-grained parts, adapt to changes, and align with task instructions.Source: §1 · p. 2
- MomaGraph unifies spatial and functional relationships with part-level interactive nodes in a compact, task-relevant scene representation.MomaGraph-Scenes supports this representation with multi-view observations, executed actions, interactive object parts, and task-aligned scene graph annotations.Source: §1 · p. 2
- MomaGraph-Bench evaluates six reasoning capabilities, from high-level task planning to fine-grained scene understanding.The benchmark addresses the lack of a unified way to assess whether task-oriented scene graphs improve planning.Source: §1 · pp. 2–3
- MomaGraph-R1 surpasses open-source baseline models, with reported gains across public benchmarks and effectiveness in real-world robotic experiments.The model is a 7B vision-language model trained with reinforcement learning to predict task-oriented scene graphs and plan from them.Source: §1 · pp. 2–3
2 RELATED WORKS
Prior work treats scene graphs as a bridge between scene understanding and action planning, but often omits either spatial structure or functional interaction. VLM planners also face grounding challenges, motivating task-specific graphs generated alongside planning.
Scene Graphs for 3D Indoor Scene Understanding
- Spatial-only graphs capture layouts but miss functional interactions, while functional graphs capture affordances but omit overall spatial structure.The paper positions MomaGraph as unifying both relation types, adding part-level nodes, and explicitly modeling state changes.Source: §2 · p. 3
Zero-shot Embodied Task Planning with VLMs
- Direct VLM planning can be sensitive to visual noise and shallow semantic grounding, while prior graph-based approaches may assume reliable graphs or separate graph construction from planning.MomaGraph’s Graph-then-Plan strategy has one VLM generate task-specific graphs before planning.Source: §2 · p. 3
3 PRELIMINARY FINDINGS AND MOTIVATION EXPERIMENTS
The motivating experiments compare direct visual planning with Graph-then-Plan and test whether unified spatial-functional graphs outperform single-relation alternatives. They report better performance for the unified representation and more complete plans from Graph-then-Plan.
3 PRELIMINARY FINDINGS AND MOTIVATION EXPERIMENTS
- The motivating experiments test direct planning reliability and whether single-relation graphs suffice for embodied agents.The evaluations use real-world household tasks and compare graph variants across base models under consistent configurations.Source: §3 · p. 3, §3.1 · p. 3, §3.2 · p. 4
3.1 ARE VLMS RELIABLE FOR DIRECT PLANNING WITHOUT SCENE GRAPHS?
- Graph-then-Plan produces correct, complete action sequences aligned with ground-truth logic, unlike direct plans that can omit prerequisites or misidentify interactions.The controlled household tasks include opening a window and obtaining clean boiled water, requiring functional, spatial, and multi-step reasoning.Source: §3.1 · p. 4, Figure 2 · p. 4
3.2 ARE SINGLE-RELATIONSHIP GRAPHS ADEQUATE FOR EMBODIED AGENTS?
- Unified spatial-functional graphs outperform spatial-only and functional-only variants across evaluations using MomaGraph-R1 and LLaVA-Onevision.The comparison keeps graph topology consistent while constraining edge types; Table 1 presents results across task tiers.Source: §3.2 · p. 4, Table 1 · p. 4
4 METHOD
MomaGraph builds task-conditioned scene graphs that combine spatial and functional relations, then uses reinforcement learning and observed state changes to make those graphs more useful for embodied planning. MomaGraph-R1 generates these graphs for Graph-then-Plan, while the dynamic update refines ambiguous object relationships after actions.
4.1 MOMAGRAPH DEFINITION
- A task-oriented graph grounds an instruction in relevant objects from multi-view images, with directed edges encoding their spatial and functional relationships.Functional edges point from the triggering object to the affected object.Source: §4.1 · p. 5
4.2 VLMS LEARN SCENE GRAPH REPRESENTATIONS WITH REINFORCEMENT LEARNING
- The graph-based reward checks action-type prediction, spatial-functional edge similarity, and completeness of task-relevant nodes.The final reward also includes format validation and a penalty for overly verbose outputs.Source: §4.2 · p. 5
- MomaGraph-R1 is a 7B vision-language model trained with reinforcement learning to generate compact, task-relevant scene graphs for embodied planning.Its graph-based reward evaluates task-oriented design principles; the model also serves as a zero-shot planner in a Graph-then-Plan framework.Source: §4.2 · pp. 5–6
4.3 STATE-AWARE DYNAMIC SCENE GRAPH UPDATE
- After an action, observed state changes refine hypothesized functional links by removing inconsistent possibilities and strengthening confirmed correspondences.For example, a knob that ignites a burner is linked to it, while ineffective knob-to-burner edges are pruned.Source: §4.3 · p. 6, Figure 3 · p. 6
5 DATASET AND BENCHMARK
MomaGraph-Scenes combines spatial and functional relations with part-level annotations in task-driven household graphs, while MomaGraph-Bench evaluates scene understanding and planning across varied capabilities and difficulty levels. The dataset and benchmark include diverse observations and tasks intended to assess reasoning beyond memorized environments.
5.1 MOMAGRAPH-SCENES DATASET
- MomaGraph-Scenes jointly annotates spatial and functional relations, including interactive parts, across about 1,050 task subgraphs and 6,278 multi-view images.Its data cover more than 350 household scenes and 93 task instructions.Source: §5.1 · pp. 6–7
5.1.1 DATASET ANNOTATION
- Annotations allow partial, diverse viewpoints and natural task instructions that may leave required objects implicit, while graphs include task-relevant objects and actionable parts.Edges capture both spatial and functional relations; the dataset defines six functional types and nine spatial types.Source: §5.1.1 · p. 7
5.2 MOMAGRAPH BENCHMARK AND EVALUATION
- MomaGraph-Bench assesses six reasoning capabilities across four difficulty tiers, from high-level planning to fine-grained scene understanding.Capabilities include action sequence, spatial, affordance, precondition-and-effect, goal-decomposition, and visual-correspondence reasoning.Source: §5.2 · pp. 7–8
- The benchmark is designed to test whether improvements in scene understanding correspond to tangible improvements in downstream planning and reasoning.Its six core capabilities are supplemented by Dynamic Verification and Long-horizon Task Decomposition tasks for temporal reasoning and multi-step planning.Source: §5.2 · p. 7, Figure 4 · p. 8
- The benchmark comprises 294 unseen indoor scenes, 1,446 multi-view images, and 352 task-oriented graphs spanning 1,315 instances from simple manipulation to multi-step replanning.It is formulated as a multiple-choice VQA task.Source: §5.2 · p. 8
6 EXPERIMENTS
The experiments assess graph-conditioned planning, visual correspondence, and real-robot task execution. MomaGraph-R1 performs competitively on benchmarks and succeeds on a complex multi-step robot task, with graph-generation and action-ordering errors remaining failure modes.
6.1.1 RESULT ANALYSIS.
- Across models, Graph-then-Plan outperforms Direct Plan, comparing planning with versus without a task-oriented scene graph.The evaluation uses multi-view observations and task instructions, with the graph-conditioned setting generating a graph before planning.Source: §6.1.1 · p. 9, §6.1 · p. 9
- MomaGraph-R1 performs on par with Claude-4.5-Sonnet and GPT-5, surpasses leading open-source VLMs, and improves +11.4% over Qwen2.5-VL-7B under w/ Graph.Closed-source models retain the highest absolute performance, while MomaGraph-R1 shows a smaller performance drop than most open-source baselines as task complexity rises from Tier 1 to Tier 4.Source: §6.1.1 · p. 9
6.2 BENCHMARK EVALUATION FOR VISUAL CORRESPONDENCE
- MomaGraph-R1 leads open-source VLMs by 3.8% on BLINK and 4.8% on the MomaGraph-Bench correspondence benchmark.The experiments report enhanced correspondence performance across open-source models when using scene graphs.Source: Table 3 · p. 9, §6.2 · p. 10
6.3 REAL ROBOT DEMONSTRATIONS
- On unseen scenes, MomaGraph-R1 generates task-specific subgraphs from active multi-view perception, then plans actions executed through parameterized primitive skills.The four demonstrations include two local interactions and two remote interactions; the skills are task-specific and derived from teleoperation data.Source: §6.3 · p. 10, Figure 5 · p. 10
6.4 QUANTITATIVE REAL-ROBOT EVALUATION
- The system achieves 70% overall success across 10 trials on a complex long-horizon task, with 80% graph-generation success and 87.5% planning success given correct graphs.Reported failure modes include spatial-relation errors or missing graph nodes, and suboptimal action sequencing.Source: §6.4 · p. 11, Figure 6 · p. 11
7 CONCLUSION
The conclusion presents MomaGraph as a unified scene representation for embodied agents, supported by a dataset, model, and benchmark. The dataset also draws on curated real-world household interactions and multi-view observations.
7 CONCLUSION
- MomaGraph unifies spatial and functional relationships with interactive elements, while MomaGraph-R1 predicts task-oriented graphs and plans zero-shot using Graph-then-Plan.The paper reports state-of-the-art performance among open-source models, competitiveness with closed-source systems, and transfer to public benchmarks and real-robot experiments.Source: §7 · p. 11
- MomaGraph-Bench evaluates fine-grained reasoning and high-level planning, addressing the stated need for systematic evaluation of task-oriented scene graphs.Source: §7 · p. 11
A.1.1 REAL-WORLD DATASET SOURCE AND COLLECTION.
- MomaGraph-Scenes combines manually collected household interaction data with curated keyframes from OpenFunGraph and SceneFun3D videos to provide multi-view RGB observations.The video sources depict human–object interactions in indoor contexts.Source: §A.1.1 · p. 17
A.1.2 SIMULATION DATA COLLECTION
The dataset supplements real-world data with AI2-THOR simulation observations from reachable viewpoints. Its task-oriented subgraphs encode relevant objects, actions, functional and spatial relations, and multi-view visual grounding.
A.1.2 SIMULATION DATA COLLECTION
- AI2-THOR samples use diverse reachable agent viewpoints and multi-view observations, with manual filtering to exclude non-interactable elements.Source: §A.1.2 · p. 17, Figure 7 · p. 17
A.1.3 DATASET ANNOTATION AND FORMAT.
- Each task-oriented subgraph is stored as JSON with task, action, category, node, and edge annotations; nodes include objects and actionable parts, and edges encode functional and spatial relations.The example connects a remote and TV through a control relation and spatial relations.Source: §A.1.3 · p. 17, Figure 8 · p. 18
- Synchronized RGB images from multiple viewpoints ground each scene's subgraphs, supporting instruction-conditioned graph prediction and multi-view reasoning.Source: §A.1.3 · p. 17
A.1.4 MULTI-ASPECT STATISTICS OF THE TRAINING DATASET
MomaGraph-Scenes contains approximately 1,050 task-oriented subgraphs and 6,278 multi-view RGB images across more than 350 household scenes and 93 task instructions. The appendix reports statistics across room, action, functional-relation, and object distributions, as well as the model's training setup.
A.1.4 MULTI-ASPECT STATISTICS OF THE TRAINING DATASET
- Approximately 1,050 subgraphs and 6,278 multi-view RGB images cover more than 350 household scenes and 93 distinct task instructions.The paper describes this coverage as varying scene layouts, object configurations, and interaction types.Source: §A.1.4 · p. 18
- Dataset statistics cover four room types, action-to-functional-category correspondence, functional-relation prevalence, and object frequency.The figures include task or action distributions across room contexts and functional-relation distributions across tasks.Source: §A.1.4 · p. 18, Figure 9 · p. 18, Figure 10 · p. 19
A.2 TRAINING DETAILS
- Training used 8× 80GB A100 GPUs for approximately 13 hours with the EasyR1 framework.The DAPO configuration is presented in Table 4.Source: §A.2 · p. 19
A.3 TRAINING CURVE
Training and validation reward curves align, with overall reward converging near 0.93 and accuracy reward near 0.9.
A.3 TRAINING CURVE
- The overall reward converges to approximately 0.93, while the accuracy reward stabilizes near 0.9.Training and validation curves closely align across metrics, indicating good generalization without significant overfitting.Source: §A.3 · p. 19
- The format reward reaches 1.0 within the first 25 steps, as the model rapidly learns valid JSON-structured outputs.Source: §A.3 · p. 19
A.4.1 BENCHMARK DESIGN
The benchmark uses manually checked, scalable VQA questions and spans household scenes and tasks from single actions to dynamic replanning.
A.4.1 BENCHMARK DESIGN
- VLM-generated multi-choice questions use scene images, state-aware graphs, and task instructions to probe spatial relations, affordances, preconditions, and goal decomposition.Source: §A.4.1 · p. 19
- Manual review refines or removes ambiguous or low-quality questions, and performance is measured by accuracy on the multi-choice VQA task.Source: §A.4.1 · p. 20
- Data combines annotated key frames from SceneFun3D videos with additional real indoor scenes across bathrooms, kitchens, living rooms, and bedrooms.Source: §A.4.1 · p. 20
- Task difficulty ranges from single actions through complementary two-step and preconditioned multi-step tasks.Source: §A.4.1 · p. 20
- Dynamic verification tasks require graph-based replanning and identifying alternative interactive objects when the target is missing.Source: §A.4.1 · p. 20
B.1 COMPARISON WITH SFT AND ICL BASELINES
The section compares reinforcement learning with SFT and ICL, tests reward-weight sensitivity, and reports real-world household task execution without task-specific fine-tuning.
B.1 COMPARISON WITH SFT AND ICL BASELINES
- RL training exceeds SFT by 3.1 on BLINK and 7.7 on MomaGraph-Bench, and exceeds ICL by 4.8 on BLINK and 11.4 on MomaGraph-Bench.Source: Table 5 · p. 22, §B.1 · p. 22
B.2 REWARD WEIGHT SENSITIVITY STUDY
- Performance varies by less than 2.4% on BLINK and 3.4% on MomaGraph-Bench across reward-weight configurations.Source: Table 6 · p. 22, §B.2 · p. 22
B.3 DETAILED REAL-WORLD DEMONSTRATIONS.
- A mobile manipulator completes everyday tasks including turning off a light, opening a microwave, turning on a TV, and opening a cabinet without task-specific fine-tuning.Source: §B.3 · p. 22, Figure 15 · p. 23