Source-linked AI summary
Scaling Manual-Grounded Appliance Manipulation with Data Synthesis and Unified Planning
Yuxing Long, Lei Kang, Ziyan Yu, Yuzheng Gao, Bin Cheng, Jiyao Zhang, Xiaoqi Li, Haolin Yang, Dongjiang Li, Hui Shen, Hao Dong
TL;DR
Manual-grounded appliance manipulation requires diverse data for long-horizon, state-dependent planning, but existing models perform poorly and training resources are limited. MAGE synthesizes such data from appliance manuals to build UseAppliance and train AppliancePlan, which achieves over 10× the best baseline on open-loop planning and outperforms baselines across RealAppliance-Bench tasks.
Problem
Manual-grounded appliance manipulation lacks large-scale, high-quality training data for long-horizon, state-dependent procedures across varied interfaces and operation logic.
Method
MAGE uses a Hierarchical Appliance Graph to synthesize manually verified grounding, planning, and closed-loop recovery data from appliance manuals, supporting the end-to-end AppliancePlan model.
Results
31.36% vs. 2.68% task success: AppliancePlan achieves over 10× the best baseline on open-loop planning and consistently outperforms baselines across RealAppliance-Bench tasks.
Takeaways & Limitations
UseAppliance and AppliancePlan provide a unified resource and model for manual-grounded appliance manipulation planning across diverse appliance categories and planning capabilities.
Abstract
from arXiv · showhide
Operating household appliances requires long-horizon planning that is state-dependent and robust to disturbances, yet existing large models fall short, as no sufficiently diverse, task-oriented dataset exists to support such planning. To bridge this gap, we propose MAGE, a scalable data synthesis pipeline that introduces a novel Hierarchical Appliance Graph (HAG) to automatically generate part grounding, long-horizon planning, and closed-loop recovery data from appliance manuals. With MAGE, we build UseAppliance, the first large-scale dataset for manual-grounded appliance manipulation planning, spanning 22 appliance categories with 89K+ part annotations, 53K+ manipulation tasks, and 33K+ closed-loop adjustment steps. Built on UseAppliance, we develop AppliancePlan, an end-to-end model for manual-grounded appliance manipulation planning. On RealAppliance-Bench, AppliancePlan with only 7B parameters achieves over 10x the best baseline on open-loop planning and consistently outperforms state-of-the-art models across all tasks. Real-robot experiments on six household appliances further confirm effective sim-to-real transfer, marking an important step toward general-purpose household robotics.
1 Introduction
Household appliance manipulation requires manual-grounded, long-horizon planning that handles diverse interfaces, state transitions, and disturbances, but existing methods lack suitable training data. The paper addresses this gap with MAGE and UseAppliance, then introduces AppliancePlan, which achieves strong benchmark and real-robot performance.
- Motivation: Appliance manipulation is harder than generic pick-and-place because appliances vary in interface layout, operation logic, and state transitions, requiring disturbance-robust, long-horizon procedures.
- Motivation: Manual consultation decomposes appliance operation into part grounding, open-loop manipulation planning, and closed-loop planning adjustment, evaluated jointly by RealAppliance-Bench.
- Motivation: Existing embodied planning models perform poorly because prior approaches simplify interactions or rely on latency-prone, error-compounding proprietary-model pipelines, reflecting a shortage of tailored training data.
- Method: MAGE uses a four-level Hierarchical Appliance Graph and autonomous agents with model-assisted annotation to synthesize grounding, long-horizon planning, and closed-loop recovery data from manuals.
- Contributions: 22 appliance categories, 89K+ part annotations, 53K+ manipulation tasks, and 33K+ closed-loop adjustment steps comprise UseAppliance, whose core annotations are human-labeled and double-verified.
- Results: 31.36% vs. 2.68% task success shows AppliancePlan’s over 10× open-loop-planning advantage on RealAppliance-Bench, while real-robot experiments confirm sim-to-real transfer.
2 Related Work
Related work on appliance manipulation spans direct device interaction and manual-based action planning, while multimodal foundation models have broadened embodied planning capabilities. Prior manual-grounded systems demonstrate the value of manuals but rely on multi-stage zero-shot pipelines with cascaded external modules.
- Direct appliance-interaction methods mainly address short-horizon behaviors, such as opening a microwave door, without manual-grounded operation.
- CheckManual and ApBot demonstrate the value of manual information for action planning but use multi-stage zero-shot pipelines with cascaded external modules.
- Multimodal foundation models have improved long-context understanding and multimodal reasoning, stimulating research on embodied planning.
- Embodied planning systems include Say-Can, PaLM-E, RT-2, and OpenVLA, which respectively combine language models with affordances, integrate multimodal perception and language, tokenize robot actions, and pursue large-scale VLA transfer.
3 MAGE: Data Synthesis Pipeline
MAGE is a scalable manual-grounded appliance data synthesis pipeline built around a Hierarchical Appliance Graph (HAG). It constructs HAGs through automated manual collection and human-verified annotation, then synthesizes grounding, long-horizon planning, and closed-loop recovery data.
- Pipeline overview: MAGE addresses the lack of large-scale manual-grounded appliance manipulation data with a four-stage pipeline centered on automated HAG construction and data generation.The first two stages build each HAG through automated collection and human-verified annotation, while the latter stages consume completed HAGs for synthesis.
- Hierarchical Appliance Graph: A HAG represents manual knowledge as a graph whose nodes span document, page, part, and state layers, while edges encode containment, reference, and state transitions.Part nodes include visual bounding boxes, state nodes represent configurations, and directed action edges encode atomic transitions between states.
- Hierarchical Appliance Graph: The HAG supports combinatorial task coverage, locally verifiable annotations, and domain-agnostic extensibility across product manuals.Sampling initial–goal state pairs yields diverse tasks, each layer can be independently audited, and the four-layer schema generalizes to any product manual.
- Observation generation and interruption injection: The completed HAG also supports part-grounding annotations and planning-aligned observation images with structured disturbances for closed-loop recovery data.Figure 2 identifies three synthesized data types: part grounding annotations, open-loop manipulation plans, and observation images with closed-loop recovery data.
- Manipulation task and planning creation: MAGE generates open-loop task–plan pairs by traversing HAG state-transition graphs and instantiating appliance-specific templates with sampled states, objects, durations, and operations.Uniform initial–goal state sampling provides systematic state-space coverage, while an LLM rewrites filled templates into semantically faithful natural-language instructions.
4 UseAppliance Dataset
UseAppliance is the first large-scale dataset for manual-grounded appliance manipulation planning, built from 22 appliance categories with verified annotations and unified supervision. It covers part grounding, open-loop planning, and closed-loop recovery while capturing long-tail parts and category-specific operation patterns.
- UseAppliance spans 22 appliance categories as the first large-scale dataset for manual-grounded appliance manipulation planning.
- High quality: Human-labeled part bounding boxes and state-transition graphs, manually authored templates, and independent verification provide high-quality annotations.Every annotation passes at least one round of independent human verification.
- Multi-task coverage: 89K+ bounding boxes, 53K+ task–plan pairs, and 33K+ recovery steps provide unified supervision for grounding, open-loop planning, and closed-loop adjustment.
- Rich diversity: Part annotations follow a long-tail distribution, with dense coverage of common parts and sparse coverage of appliance-specific controls.This distribution encourages robustness to rare structures.
- Rich diversity: 8.15 open-loop steps per task and category-dependent action distributions capture diverse operation patterns across appliances.Knob-heavy appliances are dominated by Rotate, whereas panel-based appliances rely more on Press.
5 AppliancePlan Model
AppliancePlan is a Qwen2.5-VL-7B-Instruct-based multimodal model that converts manual pages, observations, and task instructions into autoregressive outputs. It jointly trains three core manipulation capabilities and three auxiliary objectives using UseAppliance data.
- Model Architecture: AppliancePlan uses Qwen2.5-VL-7B-Instruct to encode manual pages, observation images, and textual task instructions before autoregressively predicting output sequences.Visual inputs are projected into the LLM token space and combined with tokenized task text.
- Main Objectives: The three main objectives train appliance part grounding, open-loop manipulation planning, and closed-loop planning adjustment.Grounding predicts bounding-box coordinates; open-loop planning generates ordered parameterized actions; adjustment predicts corrective actions from execution state.
- Main Objectives: Appliance part grounding maps symbolic manual part descriptions to pixel-level observation locations by predicting bounding-box coordinate sequences.Its multimodal input includes a manual image, an observation image, and a part label.
- Auxiliary Objectives: Manual-appliance part alignment strengthens bidirectional correspondence between manual parts and observations through manual-side identification and complementary alignment sub-objectives.The objective receives manual pages and an observation image as input.
- Auxiliary Objectives: The auxiliary objectives also supervise key-step action prediction, parameter VQA, remaining-step prediction, and explicit part-state judgment.Key-step supervision targets precise parameters such as rotation angles and button-press counts, while state judgment predicts ground-truth state labels.
6 Experiments
On RealAppliance-Bench, AppliancePlan substantially outperforms baselines across grounding, open-loop planning, closed-loop adjustment, and sequential planning with correction. Ablations and six-appliance robot experiments further assess the contributions and physical transfer of its unified approach.
- Appliance Part Grounding: AppliancePlan achieves 22.96% IoU and 22.24% mAP@0.5 for appliance part grounding, surpassing ApBot by 12.36 IoU and 10.14 mAP@0.5 points.The gains are attributed to explicit manual-conditioned training that links part descriptions with appliance-specific visual appearances.
- Open-loop Manipulation Planning: 47.86% task completion and 31.36% task success exceed the best open-loop baseline’s 4.36%/2.68% by over 10×, with AppliancePlan ranking first across appliance categories.Training on initial-state, goal-state, and fully parameterized plan triples addresses structured multi-step planning.
- Closed-loop Planning Adjustment: 37.12% step-wise success is highest in closed-loop adjustment, exceeding RoboBrain 2.0-7B by 5.35 points and Gemini 2.5 Pro by 5.39 points.This indicates stronger robustness to execution deviations across appliance categories.
- Sequential Planning and Adjustment: 44.59% completion and 28.07% success outperform the best baseline’s 5.84%/4.08% in sequential planning and adjustment, including recovery from model-generated errors.This end-to-end setting jointly evaluates initial planning and subsequent corrections.
- Ablation Study: Removing Manual-Appliance Part Alignment most harms grounding, Key-Step Action Prediction most harms long-horizon metrics, and Part State Judgment mainly harms closed-loop recovery.Together, the three objectives provide the best balance across grounding, planning, and correction.
- Real-Robot Evaluation: AppliancePlan is deployed on a Franka Emika Panda with RealSense D415 cameras and evaluated on six household appliances to test transfer to physical execution.The tested appliances are a microwave, air fryer, toaster, rice cooker, coffee machine, and blender; each receives a CAD/URDF model for pose estimation.